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Overview 



Does providing instruction-related professional development to school principals set in motion a 
chain of events that can improve teaching and learning in their schools? The Instructional Lead- 
ership Study provides suggestive although not definitive evidence that it does. The study exam- 
ines a theory of school change articulated by the Institute for Learning at the University of Pitts- 
burgh. The IFL provides technical assistance to school districts, primarily through strategic 
planning, coaching, and professional development for district and school administrators; it has 
also enunciated a set of “Principles of Learning” about the ideas and practices that promote stu- 
dents’ academic achievement. According to the IFL’s theory, through leadership training, 
school principals learn about high-quality instruction and about actions that they can take to mo- 
tivate and support their teachers. Principals then organize professional learning for their teachers 
and otherwise help teachers improve their classroom practices. With improved instruction, the 
theory maintains, student achievement will also improve. 

To test this theory, the researchers recruited 49 elementary schools in three districts that had 
been working with the IFL for one to five years at the time the study began. The schools were 
located in Austin, Texas; Saint Paul, Minnesota; and Region 10 in New York City (which has 
since been reorganized). By design, all the schools served large proportions of economically 
and educationally disadvantaged students, and all the schools volunteered to participate in the 
study. The study focuses on elementary schools because the researchers reasoned that the prin- 
cipal’ s role as an instructional leader would be especially pronounced in these settings. Princi- 
pals and third- and fourth-grade teachers at the schools completed surveys that asked about the 
professional development activities with which they had been involved and about other matters. 
The research team also conducted observations in some 300 third-grade reading and math 
classes, and school-level data on the achievement of third-graders came from state Web sites. 
Data were collected primarily during the 2005-2006 school year; budgetary constraints pre- 
cluded carrying out a second round of data collection that had been planned. 

Statistically significant associations connected each pair of steps in the theory of action. Thus, 
principals who received more professional development were more actively involved in the pro- 
fessional development of their teachers. Teachers who got more professional development 
taught lessons that were of higher instructional quality. And schools where instructional quality 
was higher also had students with higher academic achievement. Because the data were col- 
lected during the same time period, however, the time sequence of these phenomena cannot be 
established, and the absence of a counterfactual (evidence of what would have happened had the 
principals not received professional development in the first place) makes it impossible to con- 
clude that one event caused another. While the results of this study are promising, a more rigor- 
ous evaluation is needed to establish with greater certainty that instruction-related professional 
development for principals makes a difference for teachers and students at their schools. 
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Preface 



Schools and school districts spend considerable time and money on professional devel- 
opment for teachers and administrators, with the aim of improving classroom instruction and stu- 
dent achievement. In the context of standards-based education reform, high-quality professional 
development for principals and teachers is arguably more important than ever. However, despite 
the substantial investments currently being made in every state, too little is known about how pro- 
fessional development can improve teachers’ practice and student outcomes. 

The Instructional Leadership Study focuses on a program of leadership training for 
school principals that the Institute for Learning (IFL) at the University of Pittsburgh provided to 
three urban school districts. The study was not intended to be an evaluation of the effect of the 
IFL intervention, and it cannot be used to draw causal inferences about the role of professional 
development in improving teaching and learning. (In partnership with the American Institutes for 
Research, MDRC is currently conducting two random assignment impact evaluations of profes- 
sional development for teachers that will yield such inferences; early results from the first study 
will be published in 2008.) 

Instead, the Instructional Leadership Study examines the theory behind the IFL’s pro- 
gram. In brief, that theory holds that, through leadership training, principals leam what constitutes 
high-quality instruction. They also leam how to support their teachers in adopting better instruc- 
tional practices, both through professional development opportunities and through ongoing en- 
couragement and monitoring. Better classroom instruction, the theory maintains, will result in 
higher student achievement. 

Using survey and classroom observation data collected from more than 40 elementary 
schools in the three districts, the study develops quantitative indicators of each step in the theory 
of change and examines the statistical associations among these indicators. In so doing, it pro- 
vides some support for providing instruction-related professional development to school princi- 
pals in order to improve classroom teaching and student learning. A more rigorous study to con- 
firm these relationships is still needed, however. 



Gordon L. Berlin 
President 
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Executive Summary 



Nowhere is improving teaching and learning in America more critical than in those 
chronically low-performing schools that serve large proportions of economically disadvantaged 
and nonwhite students. This is the mission that the Institute for Learning (IFL), an arm of the 
Learning Research and Development Center (LRDC) at the University of Pittsburgh, has set for 
itself. Established in 1995, the IFL provides technical assistance to school districts, primarily in 
the form of strategic planning, coaching, and professional development for district and school 
administrators. In its work with districts, the IFL enunciates a set of “Principles of Learning” 
about the ideas and practices that lead to academic achievement for all students. 

The IFL's program of learning for district and school administrators is guided by a the- 
ory of action that is depicted in Figure ES.l. As the figure suggests, principals play a key role in 
the instructional improvement process by setting in motion a sequence of school-level behavior 
changes that make for improved teaching and learning. In the IFF’ s leadership training for prin- 
cipals — which is designed to last at least two years, with principals typically receiving between 
36 and 63 hours of training per year — principals learn about the Principles of Feaming and 
about concrete actions that they can take to motivate and support their teachers. Principals are 
expected to organize professional learning for their teachers as well as to monitor teachers’ 
classroom practices and help them incorporate new behaviors that are in accordance with the 
Principles of Feaming into their instructional repertory. With improved instruction, the theory 
holds, student achievement will improve. 

This report was not intended to be an evaluation of the impact of the IFF’ s work, which 
would have assessed whether changes in instructional quality and student achievement took 
place over and above what would have occurred without the IFF’s presence in the schools and 
districts. Instead, the analysis tests the IFF theory of action by examining the empirical relation- 
ships among measures of principal, teacher, and student behavior and performance that are as- 
sociated with each step in that theory. It finds evidence of systematic relationships that is consis- 
tent with the conceptual framework: Statistically significant linkages connect variables at each 
step in the theory of action with variables at the next step. While these findings are suggestive 
and promising, however, limitations in the research design and the data collected mean that 
these linkages cannot be interpreted as causal; on the question of causality, the results are ulti- 
mately inconclusive. 



Study Sites, Data, and Analytic Strategy 

Research for this report was conducted at 49 elementary schools in three districts, or 
“sites” — Austin, Texas; Saint Paul, Minnesota; and Region 10 within New York City — that 
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had worked with the IFL for between one and four years at the time that data collection began. 
Reflecting the shared priority placed by both the IFL and MDRC on improving educational out- 
comes for disadvantaged students, study schools were selected to include large numbers of stu- 
dents at high risk of adverse educational outcomes. Thus, between 82 percent and 87 percent of 
the students served by the study schools in each district were eligible for free or reduced-price 
lunch, and the schools also included more nonwhite students than did elementary schools in 
their districts more generally. Districts and schools participated voluntarily in the research. 

At all sites, the scores of third-graders at each study school on the statewide high-stakes 
test were used to measure student achievement in the spring of 2006; the average of third-grade 
students’ test scores from the three previous years served as a control variable in the analyses of 
the relationships between student achievement and other variables. Because of budgetary limita- 
tions, all the other quantitative data used in the report were collected during the 2005-2006 
school year. Surveys were collected from 44 principals and 274 third- and fourth-grade teachers 
in the study schools during the spring of 2006. Their primary purpose was to obtain information 
about the frequency and value of instruction-related professional development that both groups 
had received and, in the case of principals, had passed on to teachers at their schools. 

Researchers conducted observations in third-grade classrooms to ascertain the extent to 
which practices that incorporated three key IFL Principles of Learning — Accountable Talk, 
Academic Rigor, and Clear Expectations — were evident in the study schools. Two observation 
instruments that were developed by LRDC researchers — the Reading Instructional Quality 
Assessment (IQA) and the Math IQA — were adapted for use in the study. Almost all third- 
grade teachers in the study were observed once during a reading lesson and once during a math 
lesson, for a total of 151 reading observations and 151 math observations. 

Qualitative research, including interviews with high-level district officials and IFL liai- 
sons for the study districts, enabled researchers to gain a better understanding of the IFL’ s work. 
Further, case studies involving daylong visits to eight schools across the three sites helped illu- 
minate the findings from close-ended surveys. 

The analysis uses multiple regression analysis to ascertain the extent to which outcomes 
at each step of the theory of action are associated with (that is, are statistically linked to) out- 
comes at the one or two preceding steps in the theory. Data from teacher surveys and classroom 
observations at individual schools were aggregated so that, in all the regression analyses, the 
school is the unit of analysis. Since the goal of the study is to examine the nature of the relation- 
ships between the steps in the theory of action independent of other factors that may influence 
the outcomes, additional measures — principals’ length of experience, the average experience 
of the teachers at the school, and indicators for the three school districts in the study — are in- 
cluded in every analysis in order to control for the effects of these factors. 
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Key Findings 



Findings About Relationships Between Steps in the Theory of Action 

As noted above, outcomes at every step in the theory of action are linked to outcomes at 
earlier steps in statistically significant ways. With regard to principals' involvement in the pro- 
fessional development environments at their schools: 

• Greater receipt of instruction-related professional development on the part of 
principals and a greater value attached to that professional development are 
both significantly and positively associated with the principals’ involvement 
in professional development for their teachers. 

• Greater principal involvement in professional development for teachers is 
significantly and positively associated with the frequency with which teach- 
ers reported receiving professional development. 

In conjunction with one another, these findings suggest that delivering instruction- 
related professional development to principals may be an effective first step toward increasing 
opportunities for professional development offered to teachers at their schools. Principals who 
reported receiving more instruction-related professional development and valuing it more were 
more likely to organize formal professional development for their teachers and otherwise to en- 
gage with their teachers in instructional improvement efforts. In schools where principals re- 
ported greater involvement in these activities, teachers also reported receiving more profes- 
sional development; while such concurrence is to be hoped for, it is by no means assured. 

Further, relationships were observed between the professional development received by 
teachers and instructional quality: 

• In schools where teachers reported that they received more instruction- 
related professional development, researchers were more likely to observe 
higher implementation of the Principles of Learning in reading lessons. 

• Schools where teachers placed greater value on the professional development 
that they had received related to the Principles of Learning were observed to 
have higher implementation of these principles in math lessons. 

• A direct relationship was observed between the role that principals played in 
professional development for teachers related to Academic Rigor and Clear 
Expectations and higher implementation of these principles in reading les- 
sons. 
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These findings suggest that providing more professional development to teachers can 
help them improve their instructional practices, especially in reading. Furthermore, these find- 
ings provide some evidence for a direct link between principals’ involvement in professional 
development for teachers on good instructional practices and teachers’ implementation of these 
practices, at least in reading lessons. 

Finally, relationships were observed between instructional quality and student achieve- 
ment on standardized exams in math and reading: 

• Higher school-level scores on the Reading and Math IQAs are associated 
with greater percentages of students meeting the standard on the reading and 
the math state assessments, respectively. 

• In particular, higher implementation of the principles of Accountable Talk 
and Academic Rigor in reading lessons are associated with higher student 
achievement in reading, and higher implementation of the principle of Ac- 
countable Talk in math lessons is associated with higher student achievement 
in math. 

Thus, higher instructional quality is significantly related to higher student achievement 
in both reading and math. Instruction that requires students to tackle challenging tasks and to 
back up their assertions with evidence and reasoned arguments is especially associated with 
higher achievement. These findings are noteworthy because the analysis controlled for the 
achievement of prior classes of third-graders. 

Other Findings 

• Both principals and teachers especially valued opportunities to learn from 
their peers. 

• Instructional quality in third-grade reading and math classes, as measured by 
overall IQA scores and by scores on the three Principles of Learning, was 
generally low: Students were rarely pushed to analyze their work, to provide 
evidence for their claims, or to hold each other accountable during class dis- 
cussions. 

• Teachers’ reports of what they do during reading and math lessons generally 
did not match what researchers observed teachers doing. 



ES-5 




Implications and Limitations of the Study 

The analyses in the report suggest that in-service professional development for princi- 
pals can serve as a catalyst for positive changes in teaching and learning in the principals’ 
schools. While statistically significant linkages were discovered between all the steps in the the- 
ory of action, these connections must be regarded as exploratory and provisional rather than 
definitive, for two main reasons. 

First, the small number of schools and of teachers sampled within each school makes 
the quantitative estimates about the size of relationships unstable — that is, highly sensitive to 
the exact sample used in the statistical analyses. The small sample size also limits the gener- 
alizability of the findings. It is impossible to know whether the same patterns would be ob- 
served in other districts where the IFL is working, in other schools in the same districts as the 
study schools, or even in grades other than third grade within the study schools. 

Second, the theory of action informing the report assumes that behaviors will unfold in 
sequences that have an implicit temporal order. But because budgetary pressures precluded a 
second round of data collection, the data used in the study were all collected within the same 
academic year, so that is not possible to determine which actions come first and which follow. 
For this reason, the analysis can provide no empirical evidence that one action caused another. 

In fact, for most of the linkages set forth above, an alternative explanation to the one 
suggested by the IFL theory of action can be found. For example, more professional develop- 
ment in reading may be associated with higher instructional quality, but it cannot be assumed 
that the professional development led to better teaching. Rather, teachers who are already skilled 
may be especially motivated to seek professional development that will make them even better 
at what they do. 

Because the data supporting the theory of action are promising but inconclusive, a more 
definitive research project to study the impacts of professional development aligned with this 
theory is warranted. The key requirement for such a study is the presence of a counterfactual — 
an estimate of what would happen in similar schools if their principals do not receive such pro- 
fessional development — preferably a counterfactual that is established through a lottery-like 
process known as “random assignment.” If selection for the treatment is truly random and the 
sample size is reasonably large, preexisting differences between the treatment and control group 
principals and schools should be effectively eliminated, so that any differences between the two 
sets of schools that emerge over time can safely be attributed to their principals’ having received 
the professional development or not having done so. The present study, it is hoped, will lead to 
a rigorous evaluation of the theory of action guiding the IFL’s promising efforts to improve in- 
structional leadership, teaching, and learning. 
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Chapter 1 

Introduction 



The association between school leadership and student achievement is well established 
in the research literature. Reviews of quantitative studies suggest that leadership is second only 
to classroom instruction among the school-related factors that contribute to student learning. 1 
Much is known, too, about how effective principals lead: by setting goals, by improving the 
school setting, and by helping teachers develop instructional skills. 2 Finally, researchers have 
linked high-quality instruction to student achievement. 3 

What is less well understood is whether professional development offered to school 
leaders can improve low-performing schools — or the process by which such transformation 
can occur. A study of the chain of actions linking professional development for principals to 
better teaching and increased student learning is particularly timely, given the accountability 
context in which school leaders now operate. Principals are under pressure to produce results, 
especially increased test scores and reduced achievement disparities associated with income and 
race. This pressure to improve achievement places a premium on defining effective leadership 
and producing credible findings about the connections between leaders’ actions, desired instruc- 
tional practice, and student achievement. 

The Instructional Leadership Study, an exploratory effort to describe these connections 
in quantitative terms, was conducted by MDRC, a nonpartisan, nonprofit research organization 
whose mission is to build knowledge to inform educational and social policy. The study relies 
chiefly on data for the 2005-2006 school year, from multiple sources — data from principal and 
teacher surveys, classroom observations, and student scores on standardized tests, along with 
qualitative data from interviews and job-shadowing of principals in case-study schools — to 
examine the behavior of principals, teachers, and students at various steps of the school trans- 



'See Hill (1998). See also reviews by Hallinger and Heck (1996a, 1996b, 1998). The reviews conclude 
that leadership has small but important effects on student outcomes. According to Creemers and Reezigt 
( 1996), leadership accounts for one-quarter of the total variation in student learning across schools that is ex- 
plained by school-level variables (as opposed to the characteristics of the students who attend those schools). 
For a broad review of the literature on leadership and learning, see also Leithwood, Seashore Louis, Anderson, 
and Wahlstrom (2004). 

2 See Heck and Hallinger (1999); Conger and Kanungo (1998); and Leithwood, Menzies, Jantzi, and 
Leithwood (1996). While these authors have developed different vocabularies for describing what leaders do, 
the underlying concepts are similar. 

3 See Aaronson, Barrow, and Sander (2007); Borman and Kimball (2005); Resnick, Matsumura, and 
Junker (2006); and Wenglinski (2000). Each of these studies demonstrates a significant association between a 
measure of teaching practice or teacher effectiveness and student achievement on standardized examinations. 
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formation process and to describe how behaviors at these different steps are linked with one 
another. 4 This report presents the results of analyses that exploit this data set to address ques- 
tions about instructional leadership, effective teaching, and how leadership and teaching to- 
gether affect student achievement. These analyses seek to: 

• Develop reliable quantitative measures of principals’ leadership actions, 
which encompass both learning behaviors (for example, receiving profes- 
sional development focused on instructional improvement) and teaching be- 
haviors (such as being involved in professional development for teachers in 
their schools) 

• Generate reliable nonexperimental estimates of the empirical linkages among 
variables that describe (1) the actions of school leaders, (2) teachers’ receipt 
of professional development, (3) teachers’ instructional practices, and (4) 
student achievement 

The opening section of Chapter 3 addresses the first of these objectives; the concluding 
sections as well as Chapters 4 and 5 present findings that address the second goal. 

The theory of action guiding the Instructional Leadership Study was developed by the 
Institute for Learning (IFL), an arm of the Learning Research and Development Center (LRDC) 
at the University of Pittsburgh. Established in 1995, the IFL provides technical assistance to 
school districts, primarily in the form of strategic planning, coaching, and professional devel- 
opment for district and school administrators. The 25 districts with which the IFL has partnered 
over its 12-year history are primarily urban and serve large numbers of nonwhite students who 
are economically disadvantaged. The IFL's professional development translates research find- 
ings into actions that leaders can take to improve teaching and learning. Thus, its focus is on 
instruction and, specifically, on inculcating a set of ideas and practices that are associated with 
increased student achievement. 

The findings presented in this report are based on research conducted at 49 schools in 
three districts, or “sites” — Austin, Texas; Saint Paul, Minnesota; and Region 10 within New 
York City — that had worked with the IFL for between one and four years at the time that data 
collection began. 5 At the same time, however, the report should not be considered an evaluation 



4 The original plan was to collect an additional round of survey, observational, and achievement data dur- 
ing the 2006-2007 school year, but resource constraints made this plan infeasible. The decision to limit data 
analysis to one year, while unavoidable, has consequences for the analysis, as described in Chapter 2. 

5 For purposes of convenience. Region 10 in New York City is referred to as a “district” in this report, al- 
though this is obviously a misnomer. During the period of the study. New York City’s public schools were 
organized into geographical regions, each headed by a regional superintendent. The regional superintendents 

(continued) 
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of the impact of the IFL' s work. A proper evaluation of that impact would require measuring 
the outcomes that occur with and without the IFL’ s intervention in order to ascertain the differ- 
ence that the IFL’s work makes. Such an evaluation is beyond the scope of this study. 

Instead, the analysis examines whether the data from the study schools line up in pat- 
terns that are consistent with the IFL theory of action. Such alignment could not be taken as 
proof that the IFL' s work caused the patterns of behavior that the data reflect. But it would pro- 
vide empirical evidence for the theoretical framework that underlies the IFL’s efforts to improve 
teaching and learning in low-performing schools by enhancing the knowledge and skills of 
school leaders. 6 

The next section of this introductory chapter discusses the key principles that guide the 
IFL’s work and the theory of action that is explored in this study. Subsequent sections describe, 
in turn, the process for selecting the study districts and schools and the IFL’s work in the study 
districts. The chapter’s final section provides a “roadmap” for the rest of the report. 

The IFL’s Philosophy of Instructional Change and the 
Professional Development That It Offers 

In its work with districts, the IFL promulgates a set of principles about the ideas and 
practices that lead to academic achievement for all students. It also organizes a program of 
learning for district and school administrators that is intended to set in motion a sequence of 
school-level behavior changes that make for improved teaching and learning. 

Box 1.1 presents the IFL's “Principles of Learning,” a set of statements about student 
learning and the conditions that promote it. The shorthand forms of these statements (shown in 
bold type in the box) help to create a common vocabulary for talking about effective instruction. 
As one district official put it, “The Principles of Learning cover all the things that make excel- 
lent classrooms excellent. The Principles of Learning give a name to what these practices are.” 

The first Principle of Learning shown in Box 1.1 — Organizing for Effort — is argua- 
bly the most fundamental. This is the belief that students’ sustained and directed effort is more 
important to their academic success than mere aptitude — or, as one district official put it, “In- 
telligent is something you get, not something you are.” (In IFL parlance, the principle is often 



had considerably less autonomy and control over the budget than the school superintendents in Austin or Saint 
Paul, although the number of elementary schools in Region 10 (there were 49) surpassed the number in many 
medium-size districts. Beginning in the 2007-2008 school year. New York City adopted a new school organi- 
zation plan that replaced the regional organization. 

6 One recent study by IFL researchers has established a link between middle school teachers’ instructional 
quality scores and student achievement on standardized assessments (Resnick, Matsumura, and Junker, 2006). 
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Box 1.1 



The IFL Principles of Learning 

Organizing for Effort 

Sustained and directed effort, rather than aptitude, determines what and how 
much students learn. Everything should be organized to evoke and support this 
effort: Schools should set high minimum standards, teach all students a rigorous 
curriculum, and give students as much time and instruction as they need to meet 
expectations. 

Fair and Credible Evaluations 

To evoke sustained effort, schools should use assessments that students find fair 
— that are aligned with what is taught and with standards and that are graded 
against absolute standards rather than on a curve. Such assessments also provide 
parents, colleges, and employers with credible evaluations of what students know 
and can do. 

Recognition of Accomplishment 

Recognition of authentic accomplishment motivates students. Recognition can 
celebrate work that meets standards or that reaches intermediate benchmarks en 
route to the standards. Progress points should be articulated so that each student 
can meet real accomplishment criteria often enough to be recognized frequently. 

Socializing Intelligence 

Intelligence is a set of problem-solving and reasoning capabilities, along with the 
habits of mind that lead one to use these capabilities regularly; it also encom- 
passes beliefs about one’s right, obligation, and capacity to make sense of the 
world over time. Educators should “teach” intelligence by holding students re- 
sponsible for using these thinking skills. 

Self-Management of Learning 

Students need to develop and use an array of metacognitive (self-monitoring) and 
self-management strategies: noticing when they don't understand something and 
taking steps to remedy this, formulating questions that lead to deeper understand- 
ings, and judging progress toward a goal. Schools should model and encourage 
use of these strategies. 

(continued) 



4 





Box 1.1 (continued) 

Learning as Apprenticeship 

Learning environments should be organized so that teachers model complex thinking 
and analysis and so that students receive mentoring and coaching as they undertake 
extended projects. 

-Accountable Talk 

Classroom talk that promotes learning must put forth knowledge that is accurate and 
relevant to the issue under discussion, use appropriate evidence, respond to and fur- 
ther develop what others in the group have said, and follow norms of good reasoning. 
Teachers should create the norms and skills of Accountable Talk in their classrooms. 

" Academic Rigor in a Thi nk in g Curriculum 

Problem-solving and thinking should be taught in the context of a solid foundation of 
knowledge of major concepts that students are expected to know deeply. Teaching 
should engage students in active reasoning about these concepts and promote the ac- 
tive use of knowledge in every subject and at every grade level. 

* Clear Expectations 

Schools should define explicitly what students arc expected to learn. Descriptive cri- 
teria and models of work that meets standards should be publicly displayed and used 
to help students analyze and evaluate their own work. 

SOURCE: Adapted from descriptions of the Principles of Learning available on the IFL Web site (Uni- 
versity of Pittsburgh, Learning Research and Development Center, Institute for Learning, 2007). 

NOTE: Asterisks denote the Principles of Learning that are of central concern to this study. 



referred to as “Effort Creates Ability.”) A corollary of this belief is that schools and classrooms 
must be organized to guide and support student effort. The remaining Principles of Learning 
detail the specific ways that such guidance and support can occur. These include such practices 
as instituting a standards-based curriculum and assessing students’ achievement against those 
standards, assigning rigorous tasks, recognizing students’ accomplishments, encouraging teach- 



5 





ers to model thinking about complex issues, and encouraging students to become cognizant of 
what they do and don’t know so that they can take steps to fill the gaps. 

The three asterisked Principles of Learning in Box 1.1 — Accountable Talk, Academic 
Rigor in a Thinking Curriculum (Academic Rigor, for short), and Clear Expectations — are 
central to the Instructional Leadership Study. These principles represent key observable charac- 
teristics of high-quality instruction as the IFL has defined it. Researchers at the Learning Re- 
search and Development Center (LRDC) at the University of Pittsburgh developed an observa- 
tional instrument designed to ascertain the extent to which these characteristics are present in 
the classroom, and the instrument has been field-tested and refined by LRDC and IFL research- 
ers. 7 As discussed in Chapter 2, MDRC adapted this instrument, known as the “Instructional 
Quality Assessment (IQA),” for use in the Instructional Leadership Study. 

Instructional quality is also the most immediate forerunner of student achievement in 
the theory of action that animates the IFL’ s work, represented graphically by Figure 1.1. Indeed, 
the efforts to build the capacity of school system personnel — district officials, principals, and 
teachers — may all be seen as directed toward the ultimate objective of improving instructional 
practice. 

IFL’s work with districts recognizes the importance of working with instructional lead- 
ers at all levels. IFL Fellows (district liaisons) and other staff members engage with the superin- 
tendent (and his or her deputy or chief academic officer, where this position exists) to develop 
strong understanding and support for change. They work with central-office staff to create the 
capacity to support instructional improvement, and they provide training to school-building 
leadership. These activities have three main objectives: (1) to establish strong top-level support 
for the IFL that will assure those further down the hierarchy of district personnel that the initia- 
tive is a key policy priority; (2) to begin creating a common language about, understanding of, 
and focus on, good instruction; and (3) to create a culture of continuous learning at all levels of 
the system. 

A full examination of the district context is beyond the scope of the study. Rather, the 
study focuses on instructional leadership as it is exercised by principals and on the subsequent 
steps in the theory of action. For convenience, these are referred to as “Steps 1 to 5” in the text 
and figures. 



7 See Junker et al. (2006) and Matsumura et al. (2006). 
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The Instructional Leadership Study 
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As Figure 1 . 1 suggests, principals play a pivotal role in the instructional improvement 
process. The IFL’s work with principals — unlike that of some principal training institutes — 
involves all school leaders, not just new principals or principals-in- training. It is aimed at help- 
ing these principals to understand the district’s vision and to take concrete action to motivate 
and support teachers in changing their instructional practices. The leadership training is de- 
signed to occupy at least two years, with principals typically receiving between 36 and 63 hours 
of training per year, organized in several six- or seven-hour blocks. 

The training for principals covers five broad topics: implementation of district and 
school reforms; the concept of effort-based education; the Principles of Learning as a frame- 
work for studying curriculum and learning environments; coherence in curriculum, professional 
development, and assessment; and the grounding of leadership practice in evidence from as- 
sessments, student work, and other sources. Each topic occupies a number of units that include 
readings, videos, PowerPoint presentations, homework assignments, and tools that principals 
are expected to use with their teachers. Principals are also exposed to lessons in language arts 
and, to a lesser extent, mathematics — lessons in which they are the learners, so that they can 
ground the discussion of Accountable Talk, Academic Rigor, and Clear Expectations in their 
personal experience. The units are customized to fit the reform plans, curricula, and assessments 
that are in place in different districts, with the second year, in particular, being targeted increas- 
ingly toward meeting a district’s specific needs. During the first or second year, too, principals 
practice conducting “Learning Walks,” a key IFL tool for improving instruction. 8 

Because IFL staff have generally lacked the time to provide more than limited (if any) 
training directly to elementary school teachers, it is the responsibility of the principal to organ- 
ize professional learning for teachers, as well as to participate directly in such learning. The pro- 
fessional development for principals includes training and simulations on how to do this. In ad- 
dition, IFL trainers model the approaches and behaviors that principals are expected to use with 
teachers and that teachers are expected to use with students — asking the principals, for exam- 
ple, to back up their statements with evidence from readings or video clips and assigning chal- 
lenging readings and exercises. Principals are also expected to build internal standards of pro- 
fessional practice by observing teachers, inspecting student work, and helping teachers learn to 
incorporate new behaviors that are in accordance with the Principles of Learning into their in- 
structional repertory. 



During Learning Walks, participants spend five to ten minutes in each of several classrooms, looking at 
student work and talking with students and teachers. At the end of the Learning Walk, participants work with 
the leader (generally the principal) to refine observations, look for patterns within the school, and identify pro- 
fessional development needs and next steps. 




As is illustrated at the top of Figure 1.1, district-level factors (for example, leadership 
stability, curriculum decisions, hiring and compensation practices) as well as the individual 
characteristics of principals and teachers (especially their level of experience) play an important 
role in shaping instructional practices and student outcomes. However, the analyses in the fol- 
lowing chapters seek to examine the relationships among principal and teacher behaviors, in- 
struction, and student outcomes that emerge across all three study districts (rather than focusing 
on how these relationships may differ among the different districts) and across experience lev- 
els of school personnel. 9 



The Selection of Study Districts and Schools 

The original intention was to recruit 80 elementary schools in three districts to partici- 
pate in the study. MDRC and the IFL worked together to identify potential study districts. 
MDRC wanted to select districts that were ethnically and geographically diverse, in order to 
enhance the relevance of the findings for policymakers and practitioners. IFL staff members 
suggested districts that met these criteria and that they thought might be amenable to participa- 
tion in a research effort. 10 

Early on, the decision was reached to limit the study to elementary schools. The re- 
search team reasoned that, in elementary schools, the role of principals as agents of instructional 
improvement is likely to be most pronounced. (In contrast, in middle and high schools, the de- 
partmentalized nature of instruction and the presence of assistant principals, deans, and depart- 
ment heads make for much more distributed instructional leadership.) The researchers also 
elected to focus on third grade, for two main reasons. First, because that is the earliest grade at 
which all students’ reading and math skills must be tested if states are to be in conformity with 
the requirements of the No Child Left Behind Act of 2001, achievement data were certain to be 
available for students at this grade level. Furthermore, third-grade students are expected to 
grapple with more complex texts and mathematical concepts than are taught in earlier grades, 
making instruction that incorporates the Principles of Learning both more relevant to and (pre- 
sumably) more evident in third-grade lessons than in lessons for younger pupils. Fourth-grade 
teachers were also asked to complete teacher surveys, but they were not included in the class- 
room observations; nor were student achievement data collected for fourth-graders. 11 



9 Statistically, this is achieved by treating district, principal’s prior experience, and teacher’s prior experi- 
ence as covariates in regression and correlational analyses. See the discussion in Chapter 2. 

10 Districts whose work with the IFL had been studied previously were excluded from consideration for 
this study, to prevent “research overload” in those districts. 

"An important part of the survey concerned teachers’ views of their principal as an instructional leader, 
and the researchers wanted to base leadership ratings on more than the handful of responses that would be pro- 

( continued) 
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From the outset, it was also determined that participation in the research on the part of 
school districts, and of individual schools within those districts, would be entirely voluntary. In 
presentations about the study to potential participants, the study director emphasized that the 
research would yield valuable information and would impose only a minor burden on teachers 
and principals. At the same time, participation in the study did not particularly benefit a district 
— the study was too leanly funded to offer big financial incentives, and the assistance that dis- 
tricts received from the IFL was not linked to joining the research. In all, seven districts were 
approached, and Austin, Saint Paul, and Region 10 of New York City agreed to participate. 12 

By design, study schools were selected to include large numbers of students at high risk 
of adverse educational outcomes. This choice reflects the shared priority placed by both the IFL 
and MDRC on improving educational outcomes for disadvantaged students. Researchers used 
2004 data to identify schools eligible for inclusion in the study. 13 Such schools had to serve a 
student population that was at least 50 percent economically disadvantaged (as measured by 
eligibility for free or reduced-price lunch) and at least 50 percent nonwhite. The pool was fur- 
ther narrowed to include the schools with lower achievement levels: 47 schools in Austin, 42 
schools in Region 10, and 29 schools in Saint Paul. 

The participation of schools, like that of districts, was wholly voluntary, and recruiting 
the schools required intensive effort on the part of research team members. While a letter from 
the superintendent at each site to the principals of the potential study schools described the 
study and invited schools to participate, meetings with the principal and/or teachers at the indi- 
vidual schools proved essential for getting schools to sign up. In the end, 49 of the 118 schools 
in the eligible pool agreed to participate. 14 



vided by a survey that was limited to third-grade teachers. At the same time, resources were insufficient to al- 
low all teachers in the school to be surveyed. Expanding the survey sample to include fourth-grade as well as 
third-grade teachers was an effoit to compromise between research desiderata and resource constraints. 

12 Two districts declined because they did not want to create any additional work for teachers, who, accord- 
ing to district administrators, were already stretched to the limit. In a third district, it proved impossible to reach 
a budgetary agreement with the district’s research office. The fourth district was undergoing a leadership tran- 
sition when the research project was initially approved, and when the new superintendent assumed office, he 
announced that the district would not continue its professional development contract with the IFL. Under these 
circumstances, it made no sense for the research to go forward in the district. 

13 These data were posted on the Web site www.SchoolMatters.com, which was created by the School 
Evaluation Services division of Standard & Poor’s as an independent, objective source of school and district 
data. To select eligible Region 10 schools, researchers also examined data posted on the New York City De- 
partment of Education Web site. 

14 For the school to be counted as participating, the principal and at least two third-grade teachers had to 
sign and submit Informed Consent Forms. Although consent was not required from all the third-grade teachers 
in a given school for the school to be considered a study school, with only one exception, all third-grade teach- 
ers in the 13 Saint Paul study schools agreed to participate. In Austin, however, there were 91 third-grade 

(continued) 
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Table 1.1 shows selected characteristics of the study schools and of all elementary 
schools in their respective sites as of the 2003-2004 school year, when the schools were se- 
lected. As planned, the study schools in all three districts contained higher proportions of low- 
income students than did other schools in their districts: Between 82 percent and 87 percent of 
the students in the study schools were eligible for free or reduced-price lunch, compared with a 
range of 65 percent to 76 percent for all elementary schools in the districts. The study schools 
also included more nonwhite students than did elementary schools in their districts more gener- 
ally. That said, the ethnic composition of students attending the study schools differed consid- 
erably among the three districts. In Austin, study school students were predominantly Hispanic; 
in Region 10, the student population was almost evenly divided between African-American and 
Hispanic students; and the Saint Paul students came from a variety of backgrounds (including a 
sizable proportion of Asian students, primarily Hmong in extraction). Finally, and also as 
planned, the study schools had higher proportions of students scoring below standard on state or 
district tests. 15 



The IFL’s Work with the Study Districts 

The IFL began its work with Saint Paul in the 2000-2001 school year; with Austin, in 
the 2001-2002 school year; and with Region 10, in the 2004-2005 school year. Typically, the 
site contracts called for 25 to 30 days of work with the site each year. The first year in all three 



teachers across the 21 study schools; of these, only 73 (80 percent) agreed to participate. The data for calculat- 
ing the participation rates of third-grade teachers in Region 10 are not available. 

While the participation of fourth-grade teachers was sought as well, their participation was not required, 
and, at four schools included in the study, only third-grade teachers signed the forms signaling participation. 

1 5 A much higher proportion of students in Austin than in the other districts met the state standard. The size 
of the disparity strongly suggests that, at least in 2004, standards in Texas were easier to satisfy than in the 
other two districts. Evidence from the 2005 National Assessment of Educational Progress (NAEP, often re- 
ferred to as “the Nation's Report Card”) supports this proposition. Fourth-grade students in New York State 
who were eligible for free or reduced-price lunch had an average NAEP reading score of 210.3; in Minnesota 
and Texas, the corresponding scores were 208.6 and 207.8, respectively (U.S. Department of Education, Na- 
tional Center for Education Statistics). As Table 1.1 shows, roughly similar proportions of students at the study 
schools in the three participating districts qualified for free or reduced-price lunches (between 82.4 percent and 
87.1 percent). Nonetheless, only 15.9 percent of Austin third-graders in the study schools scored below stan- 
dard on the state or district reading test, compared with 73.5 percent in Region 10 and 57.5 percent in Saint 
Paul. 

Given the educational and economic disadvantages faced by many study school students, it is perhaps sur- 
prising that relatively few of these schools received funding from Reading First, the major federal source of 
funding for reading instruction. One possible explanation is that 2003-2004 was the first full academic year in 
which Reading First funds were available to districts; schools that did not receive funding initially may have 
done so subsequently. Another reason may be that all three study districts have adopted a Balanced Literacy 
approach that is not fully compatible with the heavy phonics emphasis of Reading First. 
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The Instructional Leadership Study 
Table 1.1 

Selected Characteristics of Study Schools and of All Elementary Schools 
in the Study Districts (2003-2004)“ 

New York City 

Austin Region 10 St. Paul 

All All All 

Study Elementary Study Elementary Study Elementary 



Characteristic Schools Schools Schools Schools Schools Schools 



Number of elementary schools 


21 


74 


15 


49 


13 


61 


Reading First schools (%) 


28.6 


23.0 


0.0 


12.2 


23.1 


8.2 


Average student enrollment 


565 


576 


627 


752 


395 


357 


Race/ethnicity (%) 


White, non-Hispanic 


6.6 


24.9 


1.9 


7.6 


16.1 


29.8 


Black, non-Hispanic 


12.6 


14.2 


49.1 


38.8 


36.2 


32.4 


Hispanic 


80.2 


58.1 


47.3 


51.2 


12.4 


11.6 


Asian 


0.5 


2.6 


1.4 


2.0 


31.3 


24.1 


Other 


0.2 


0.2 


0.4 


0.4 


4.0 


2.1 


Students eligible for free or 


reduced-price lunch 1, (%) 


87.1 


65.0 


82.4 


76.0 


84.2 


69.2 


Third-grade students scoring 
below standard on the 
state/city achievement test c (%) 


Reading 


15.9 


12.3 


73.5 


63.0 


57.5 


46.8 


Math 


18.4 


14.7 


60.6 


52.1 


55.2 


46.4 



SOURCES: MDRC calculations from publicly available data from the 2003-2004 Common Core of Data; the 
2003-2004 New York School Report Cards; the 2003-2005 Southwest Educational Development Laboratory; 
and the 2004 administrations of the Texas Assessment of Knowledge and Skills (TAKS), the New York City 
CTB-Reading and CTB-Math tests, and the Minnesota Comprehensive Assessment (MCA). 



NOTES: Rounding may cause slight discrepancies in calculating sums and differences. 

“Characteristics are reported for all public elementary schools that include grades 3 and 4. 
b For New York City Region 10 schools, only the percentage of students eligible for free lunch is reported. 
l No statewide test was administered to third-grade students in New York City in 2004, so data from the 
citywide standardized assessment are used. Eleven schools in St. Paul are missing student achievement data 
due to the suppression of statistics for schools where seven students or fewer were tested. 
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sites entailed work with a leadership cadre of key central staff — area supervisors, directors of 
special education and of the district’s initiative for English Language Learners, and so on — 
and with other leaders, to expose them to the Principles of Learning and to Learning Walks. 16 

After that, the paths of the three districts diverged. Lrom the outset, all elementary 
school principals in Austin and Region 10 were given the message that the ILL’s work would 
serve as the cornerstone of the district’s educational reform efforts and that all schools would be 
expected to participate in the professional development that the ILL provided. This was not the 
case in Saint Paul, where schools joined the ILL professional development effort in phases, over 
a multiyear period and on a voluntary basis. This decision reflected the desire of high-ranking 
district officials to try out the intervention in a few Saint Paul schools and “get it right” before 
implementing it more widely, as well as their belief that the district did not have enough 
coaches to support all schools at once; it also took into account that Saint Paul schools and their 
leaders exercised considerably more autonomy than in the other sites. (Once schools did sign 
up, however, they were expected to participate fully.) Seven elementary school principals par- 
ticipated in the professional development in the first year that it was available (2001-2002), and 
1 1 more joined in the second year. By the third year, 30 elementary school principals were re- 
ceiving ILL professional development, and, by the fourth year, all schools had joined. Gradually 
and over time, the ILL’s work became accepted as the reform model for the district. As one 
Saint Paul Public Schools staff member put it, “Now [we're] all on board with ILL — everyone 
talks the talk, and there is no doubt that this is the [district] reform.” 

As a rule, at the beginning, the ILL provided five or six full days of training to elemen- 
tary school principals and other key school-level personnel (such as assistant principals and lead 
teachers) over the course of a school year. Thereafter, the amount of ILL training provided to 
principals declined at all three sites. In one district, this was the result of a funding cutback. The 
reduction also reflected new ILL priorities; the institute had developed a new instructional ap- 
proach known as “Disciplinary Literacy,” for high school teachers, the ILL shifted much of its 
professional development to this new initiative. 17 

As in other districts with which the ILL works, principals at the study sites have been 
expected to relay what they have learned at the ILL training sessions to their faculties. 18 District 



16 The IFL liaison's work has involved training a core group of individuals from the district who could 
train others and ultimately take over from the IFL. Thus, in Region 10, for instance, instructional coaches re- 
ceived training on the Principles of Learning from district staff rather than from the IFL liaison. 

n The key premise of Disciplinary Literacy is that instruction should help students develop the “habits of 
mind” and practice the forms of inquiry that are exhibited by mathematicians, scientists, historians, and profes- 
sionals in other fields. 

ls ln Austin, the IFL liaison recognized a need to work with teachers — especially teachers new to the dis- 
trict and bilingual teachers who initially did not see the IFL’s work as relevant to their own. The liaison also 

(continued) 
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officials and IFL staff recognized that, as one district-level informant put it, “This only worked 
as well as the principal.” Another official acknowledged that the principal’s lack of capacity as a 
teacher of teachers could be a “stumbling block” but added that new principal candidates were 
now being interviewed about their abilities in this area and about their adherence to the Princi- 
ples of Learning. 

Neither district officials nor IFL staff wanted or expected the institute’s professional 
development to be the sole kind of training offered to principals and teachers. Rather, because 
IFL personnel believed that it was essential to ground the Principles of Learning in actual sub- 
ject-matter content, both district and IFL personnel envisioned the IFL's work as providing an 
“umbrella” — a set of overarching ideas about what constitutes high-quality teaching and learn- 
ing and a vocabulary for discussing those ideas — under which other professional development 
efforts could be fitted. In all three sites, subject-specific professional development directed to- 
ward teachers rather than principals occurred simultaneously with IFL-provided professional 
development. All three districts have adopted approaches to teaching literacy and math that are 
seen as consistent with the IFL Principles of Learning — and have contracted for subject- 
specific professional development that utilizes these approaches. Thus, from the outset, teachers 
in the Saint Paul schools who were engaging with the IFL simultaneously received separate 
training in writing instruction, and later in reading instruction as well, from outside consultants. 
In Austin and Region 10, teachers have received training in reading and in math from other or- 
ganizations; in Austin, training in science and in the use of formative assessments to guide in- 
struction have also been added to the mix. In general, both IFL staff and district personnel who 
were interviewed asserted that the professional development that teachers received comple- 
mented and mutually reinforced the IFL’s messages. 



The Contents of This Report 

The remainder of this report consists of five chapters. Chapter 2 lays the foundations for 
the substantive findings in Chapters 3 through 5. It begins by reviewing the data collected for 
the study and the analytic constructs created using these data. It then considers the regression 
analysis methodology used to describe in quantitative terms the relationships between con- 
structs at the various steps of the theory of action. The chapter’ s final section discusses the limi- 
tations of the analyses and the conclusions that the analyses support. 

Chapters 3 through 5 cover successive steps in the IFL theory of action. Chapter 3 fo- 
cuses on Steps 1 to 3 of that theory; its subject is the professional development that principals 



had more time available in her contract to undertake professional development for teachers than did her coun- 
terparts at the other two study sites. 
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receive and transmit to teachers in their schools and the professional development that teachers 
report receiving. First, for each step of the theory, it describes the behavior of principals who 
received higher, medium, and lower ratings as instructional leaders from teachers at their 
schools; then it employs regression analysis to link quantitatively the behaviors of principals 
and teachers with respect to professional development. 

Chapter 4 deals with Steps 2 to 4 of the theory. It delves deeply into classroom observa- 
tion data to determine the extent to which the instruction that was observed conforms to IFL 
principles. It also makes use of multiple regression analysis applied to data from the teacher 
survey to relate instructional practices to the professional development that the teachers reported 
receiving and to the involvement of principals in the professional development of their teachers. 

Chapter 5 covers Steps 3 to 5 of the theory of action. It examines the ultimate outcome of 
interest — student achievement — and its relationship to the quality of classroom instruction. The 
chapter also considers whether more professional development resulted in higher achievement. 

Chapter 6 discusses the overall findings of the study and their implications for efforts to 
improve schools. 
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Chapter 2 



The Report’s Data, Key Constructs, and 
Analysis Strategy 

This chapter first discusses the quantitative and qualitative data collected for the In- 
structional Leadership Study. It then presents the key constructs and measures that are associ- 
ated with each step of the theory of action developed by the Institute for Learning (IFL) and 
describes the general strategy that is used in Chapters 3 through 5 to analyze these data. These 
methodological sections of the report may be of interest to researchers and others, but readers 
who elect to skip them are strongly encouraged to turn to the final section of the chapter, which 
considers the limitations of the analyses and of the conclusions that can be drawn. 



The Data and Their Sources 

Table 2.1 displays the quantitative data sources used in the study. These data, along 
with the qualitative data used to provide additional meaning and context to the statistics, are 
described below. 

Principal and Teacher Surveys 

Surveys were administered during the spring of 2006 to principals and to participating 
third- and fourth-grade teachers in the 49 elementary schools in the study. 1 Three districts, or 
“sites,” were involved: Austin, Texas; Saint Paul, Minnesota; and Region 10 in New York 
City. 2 The surveys were self-administered and took about 45 minutes to complete. At each 
school, a staff member was appointed as a survey liaison whose main responsibility was to col- 
lect the completed surveys (which respondents had sealed into envelopes) and send them to the 
firm contracted by MDRC to handle survey administration. The process yielded response rates 



'The surveys used in this study are based on surveys developed by RAND Corporation for a study of the 
IFL’s work with three urban school districts (Marsh et al., 2005). An earlier round of surveys was administered 
in the fall of 2005, asking principals and teachers to reflect on the 2004-2005 school year. Because of resource 
constraints, these surveys were not analyzed. 

2 For convenience. Region 10 is referred to as a “district” in this report, although this is obviously a mis- 
nomer. During the period of the study. New York City schools were organized into geographical regions, each 
headed by a regional superintendent. Beginning in the 2007-2008 school year. New York City adopted a new 
school organization plan that replaced the geographical regions. 
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The Instructional Leadership Study 

Table 2.1 



Quantitative Data Associated with Steps in the Theory of Action 
Guiding the Instructional Leadership Study 



Step in Theory of Action 


Data Source 


Comment 


Principal's receipt of PD 


Principal surveys 


Surveys were administered to principals of the 49 study 
schools in the spring of 2006 tapping their experiences dur- 
ing the 2005-2006 school year and the summer of 2005. 
Surveys were received from 44 principals, for a response 
rate of 89.8 percent. 


Principal's involvement in 
PD for teachers 


Principal surveys 


See the comment above. 


Teacher' s receipt of PD 


Teacher surveys 


Surveys were administered to 285 third- and fourth-grade 
teachers in the 49 study schools in the spring of 2006 tapping 
their experiences during the 2005-2006 school year and the 
summer of 2005. Surveys were received from 274 teachers, 
for a response rate of 96 percent. 


Characteristics of class- 
room instruction 


Classroom observations, 
teacher surveys 


Observations were conducted of 151 third-grade reading 
classes and 151 third-grade math classes during the 2005- 
2006 school year. 


Student achievement 


State achievement tests. 
New York City test 


School-level third-grade reading and math scores on stan- 
dardized assessments administered in 2003, 2004, 2005, and 
2006 were collected. 



NOTE: For an overview of the IFL’s theory of action, see Figure 1.1 in Chapter 1 and the section entitled “The 
IFL’s Philosophy of Instructional Change and the Professional Development That It Offers.” 



of 90 percent for the principals and 96 percent for the teachers — extraordinarily high rates for 
mail surveys. 3 

Box 2.1 shows the topic areas covered in each survey. While the scope of the surveys was 
broad, their primary purpose was to collect information about the instruction-related professional 
development that both groups had received and, in the case of principals, passed on to teachers at 
their schools. Thus, a number of survey items asked about the frequency of professional develop- 



3 Teachers who completed the survey received a $15 gift card from a national bookstore chain as a token of 
appreciation. The survey liaison received a $30 gift card. 
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ment activities that centered on IFL concepts and on other instruction-related concerns. 4 (Profes- 
sional development is abbreviated as “PD” in many of the report’s exhibits.) 

Respondents were also asked to rate the value of the professional development that they 
had received on each topic. 

All the questions asked about events and respondents’ reactions during the 2005-2006 
school year and the preceding summer. 

Classroom Observations 

Researchers conducted observations in third-grade classrooms to ascertain the extent to 
which practices that incorporated three key IFL Principles of Learning — Accountable Talk, 
Academic Rigor, and Clear Expectations (see Box 1.1 in Chapter 1) — were evident in the 
study schools. 5 

As mentioned in Chapter 1 , two instruments that were developed by researchers at the 
Learning Research and Development Center (LRDC) of the University of Pittsburgh — the 
Reading Instructional Quality Assessment (IQ A) and the Math IQA — were adapted for use 
in the study. 6 Before field research began, IFL staff members provided MDRC research team 
members with five days of training on the use of the instruments. Researchers using the 
IQA take a running record of a lesson and then use their notes to assign scores to specific 
rubrics measuring the extent to which the lesson reflects Accountable Talk, Academic 
Rigor, and Clear Expectations. The point range for each rubric is 1.0 to 4.0 (with 4.0 always 



4 The surveys asked about the IFL Principles of Learning — Accountable Talk, Academic Rigor, and Clear 
Expectations — but did not explicitly use these terms. Although one goal of the IFL’s work with districts is for 
these terms to become part of a shared vocabulary for communicating about instructional quality, the research- 
ers suspected — and, to some extent, the limited field research conducted for the study has confirmed — that 
familiarity with the name of a principle did not necessarily indicate familiarity with the underlying concept. 

That said, the alternative language that the researchers developed for describing the Principles of Learning 
may not have fully captured the complex thinking associated with each principle. For example, instead of ask- 
ing about professional development on Accountable Talk, the survey asked about professional development on 
“engaging students in whole-group discussions to promote learning.” Teachers may have interpreted this ques- 
tion as tapping professional development on increasing students' participation and engagement in classroom 
discussions — necessary but not sufficient for the successful implementation of Accountable Talk. 

5 As Box 2. 1 indicates, several items in the teacher survey also asked respondents to describe their class- 
room practices; Chapter 4 discusses how well teachers’ views of their instruction corresponded with observers’ 
ratings. 

6 Both the Reading and Math IQ As were adapted for this study in an attempt to achieve greater reliability 
across raters. The versions of the Reading and Math IQAs that were used in this study are presented in Appen- 
dix A. 
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Box 2.1 



Key Topic Areas Covered in the Principal and Teacher Surveys 
Principal survey 



• Frequency with which the school or district offered teachers professional devel- 
opment on specific topics involving specific types of activities 

• Principal’s role in professional development that the school or district offered to 
teachers 

• Proportion of teachers who typically participated in professional development on 
specific topics/ involving specific types of activities 

• Frequency with which the district offered the principal professional development 
on specific topics/involving specific types of activities 

• Flow valuable each topic or type of professional development received was for 
the principal’s practice as a school leader 

• Value of IFL-related professional development 

• Importance of observing specific kinds of instructional activities, methods, and 
materials in classroom visits 

• Opinions about the reading and math curricula 

• Time spent on school leadership activities 

• Opinions about the principal’s supervisor and about the district’s practices in 
evaluating and supporting principals 

• Perceived challenges to the principal’s efforts to improve teaching and learning 

• Length of experience as a principal in the study school and in total 

Teacher survey 

• Frequency with which the teacher participated in district- or school-sponsored 
professional development on specific topics/involving specific types of activities 

• Flow valuable each topic or type of professional development received was for 
the teacher’s professional growth 

• Features of the teacher’s instructional practice during a typical reading or math 
lesson 

• Opinions about the reading and math curricula 

• Actions of the principal as a school leader and attitudes toward the principal 

• Perceived challenges to improving teaching and learning in the classroom 

• Opinions about district administrators’ practices and priorities 

• Length of experience as a teacher in the study school and in total 

• Teaching credentials (highest degree received, type of certification held) 
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indicating greater prevalence of the Principle of Learning), except for the rubric measuring 
student participation in classroom discussions, which has a range of 1.0 to 5.0. Subscores 
are calculated for each Principle of Learning, based on the scores that teachers received on 
the relevant rubrics, and the overall IQA score is calculated as the mean of the subscores for 
each Principle of Learning. 7 Details regarding the individual rubrics and how each subscore 
is calculated from these rubrics are discussed in Chapter 4. 8 The versions of the Reading and 
Math IQAs employed in the study appear in Appendix A. 

Almost all third-grade teachers in the study were observed twice during the 2005-2006 
school year — once during a reading lesson and once during a math lesson — for a total of 15 1 
reading observations and 151 math observations (97 percent of the observations that were 
scheduled for fielding). 9 The observations were spread across the school year so that, to the ex- 
tent possible, teachers who were observed earlier in the year in one subject would be observed 
later in the year in the other subject. The same purposeful scheduling of observations happened 
at the school level, so that any given school was likely to have some earlier and some later ob- 
servations in each subject area. 10 Each observation generally lasted between 45 and 90 minutes. 

Student Achievement Data 

In all three sites, the percentage of each school’s third-grade students who met the stan- 
dard on the statewide standardized test was downloaded from the state or local education 
agency’s Web site and was used to measure student achievement in 2006. Test score data from 



7 If a subscore is missing for any Principle of Learning, an overall IQA score is not calculated. 

8 As noted above, research staff members received common training on the use of the IQA. In an effort to 
achieve reliability across raters, staff also discussed — in lengthy weekly telephone conferences — general 
scoring practices as well as how to score specific problem cases. In the end, however, it proved impossible to 
attain the requisite degree of consistency among the four observers. Reliability was achieved because one 
member of the research team, on the basis of the running record, rescored all observations from Austin and St. 
Paul to be consistent with scoring standards developed by the two researchers conducting observations in New 
York City Region 10. 

9 Since 155 third-grade teachers participated in the study, in theory, 310 observations could have been 
completed (155 each in math and reading). However, some teachers could not be observed twice, for such lo- 
gistical reasons as scheduling conflicts, maternity leave, or having left the school. In addition, a few teachers 
were observed twice in one subject area because they did not teach the other subject area. 

As discussed below, some observations were missing so much data — generally because key activities 
called for by the IQA did not take place during the lesson — that they could not be used in analyses. 

10 Such scheduling was intended to ensure that not all observations for a particular school would be com- 
pleted early or late during the year, given the changes in classroom dynamics that can be expected to take place 
over time. Although MDRC is based in New York City and Oakland, California, the study employed field 
researchers in Austin and Saint Paul so that observations could be conducted over the course of the school year 
in these districts. 
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the three previous years were also downloaded for use as control variables in the analyses of the 
relationships between student achievement and other variables. 

In Texas, the statewide achievement test is the Texas Assessment of Knowledge and 
Skills (TAKS). Since the TAKS has been in use since 2003, third-grade TAKS scores were 
downloaded from the Texas Education Agency’s Web site for the 2005-2006 school year and 
for the three preceding school years (2002-2003, 2003-2004, and 2004-2005). 11 

In New York City Region 10, third-grade student achievement for the 2005-2006 
school year was measured using the New York State Test; the data were downloaded as school 
grade-level files from the New York State Education Department. 12 Since the New York State 
Test was not administered to third-grade students prior to the spring of 2006, student achieve- 
ment for school years 2002-2003 through 2004-2005 was measured using New York City CTB- 
Reading and CTB -Mathematics test data available from the New York City Department of 
Education. 13 

In Saint Paul, third-grade student achievement in the 2005-2006 school year was meas- 
ured using the Minnesota Comprehensive Assessment, Series II (MCA-II). For school years 
2002-2003 through 2004-2005, student achievement was measured using an earlier (and non- 
comparable) version of the test, simply called the Minnesota Comprehensive Assessment 
(MCA). Data from both the MCA-II and the MCA were downloaded as school grade-level files 
from the Minnesota Department of Education. 14 

Qualitative Data 

To gain a better understanding of the IFL’s work and its influence on sites and schools, 
MDRC researchers interviewed high-level officials and the IFL liaisons for the three study dis- 
tricts. Further, case studies involving daylong visits to eight schools across the three sites helped 
illuminate the findings from close-ended surveys. During these site visits, researchers inter- 
viewed and job-shadowed the principal and generally conducted a focus group with teachers. 
Finally, MDRC researchers in Austin and Region 10 observed several training sessions for 
principals and teachers at which IFL concepts were presented and discussed. 



1 'Texas Education Agency, Student Assessment Division (2006). 

l2 New York State Education Department; Elementary, Middle, Secondary and Continuing Education; In- 
formation and Repotting Services (2006). 

13 New York City Department of Education, Department of Assessment and Accountability (2006). 
l4 Minnesota Department of Education (2006). 
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Constructs and Measures Used in the Report 

Box 2.2 presents a list of the key constructs and measures used in this report; Appendix 
Table B.l reports the salient statistical properties of these variables. The measures related to 
professional development are all drawn from the principal and teacher surveys. Some measures 
consist of a single survey item; others involve scales created from multiple items. 15 Appendix C 
presents the constructs and measures that are derived from the surveys and the specific items 
that compose these measures, which are referred to repeatedly in the chapters that follow. 



The Data Analysis Strategy 

Originally, the research team had intended to conduct analyses in which the statistical 
relationships among constructs at all five steps in the IFL theory of action could be simultane- 
ously linked (see Figure 1.1 in Chapter 1). In such analyses, every construct (except those at 
Step 1) would be considered to be dependent on constructs at the preceding step(s), and every 
construct (except those at Step 5) would be considered to be a potential predictor of constructs 
at subsequent steps. 

Complex five-step models, however — especially those that include more than one 
construct at each step — require data from many more schools than were available in the study. 
For this reason, the theory is divided into four separate sets of analyses, one for each step in the 
theory that could be influenced by preceding steps in the theory. In each set of analyses, direct 
relationships are examined between the outcome of interest and the one or two steps in the the- 
ory that immediately precede that outcome. 16 



15 To derive the scales, researchers first identified survey items that they hypothesized to be associated with 
certain constructs. Factor analysis was then used to refine and improve the scales. The scales vary widely in the 
number of items they contain, from a minimum of 2 to a maximum of 37 ; the average number of items per 
scale across the 23 scales is 10. With respect to missing data, the researchers, with some exceptions, followed a 
decision rule to suppress the computation of a scale score for a respondent if more than 25 percent of the items 
making up the scale had missing data. Only survey constructs with alphas of 0.65 or higher were included in 
the analyses. 

l6 The research team also sought to explore the indirect relationships among variables, using path analyses 
in which constructs representing three consecutive steps are included in the same model. In this way, it would 
be possible to examine, for example, how constructs at Step 1 affected outcomes at Step 3 both direcdy and 
indirectly by affecting outcomes at Step 2 that then affected the Step 3 outcomes. However, because of the 
small sample size, these three-step models proved unstable, and the results are not reported here. 
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Box 2.2 

Key Constructs and Measures Used in This Report, 
by Data Source and Step in the Theory of Action 

Principal survey 

Principal’s receipt of professional development 

Frequency of principal’s receipt of instruction-related PD 
Value principal places on instruction-related PD 
Frequency of principal’s receipt of PD on Principles of Learning 
Value principal places on PD on the Principles of Lear ning 
Frequency of principal’s receipt of PD on Accountable Talk 
Value principal places on PD on Accountable Talk 
Frequency of principal’s receipt of PD on Academic Rigor 
Value principal places on PD on Academic Rigor 
Frequency of principal’s receipt of PD on Clear Expectations 
Value principal places on PD on Clear Expectations 

Principal’s involvement in professional development for teachers 

Role principal plays in PD offered to teachers 
Time principal spends with teachers on instructional improvement 
Role principal plays in PD on the Principles of Learning offered to teachers 
Frequency of principal’s giving teachers suggestions on implementing the 
Principles of Lear ning 

Role principal plays in PD on Accountable Talk 

Frequency of principal’s giving teachers suggestions on Accountable Talk 
Role principal plays in PD on Academic Rigor 

Frequency of principal’s giving teachers suggestions on Academic Rigor 
Role principal plays in PD on Clear Expectations 

Frequency of principal’s giving teachers suggestions on Clear Expectation 

Other measures 

Principal’s assessment of the school’s PD environment for teachers 



(continued) 
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Box 2.2 (continued) 



Teacher survey 

Teacher’s receipt of professional development 

Frequency of teacher’s receipt of instruction-related PD 
Value teacher places on instruction-related PD 

Frequency of teacher’s receipt of PD related to the Principles of Learning 
Value teacher places on PD related to the Principles of Lear ning 
Frequency of teacher’s receipt of PD related to Accountable Talk 
Value teacher places on PD related to Accountable Talk 
Frequency of teacher’s receipt of PD related to Accountable Talk in math 
Value teacher places on PD related to Accountable Talk in math 
Frequency of teacher’s receipt of PD related to Academic Rigor in math 
Value teacher places on PD related to Academic Rigor in math 
Frequency of teacher’s receipt of PD related to Academic Rigor in reading 
Value teacher places on PD related to Academic Rigor in reading 
Frequency of teacher’s receipt of PD related to Clear Expectations 
Value teacher places on PD related to Clear - Expectations 

Other measures 

Teacher’s assessment of the principal as instructional leader 
Teacher’s assessment of the principal as organizational leader 

Instructional Quality Assessment (IQA) 

Reading IQA score 

Accountable Talk 
Academic Rigor 
Clear Expectations 

Math IQA score 

Accountable Talk 
Rigor of the Task 
Clarity and Detail of Expectations 

Student achievement 



Percentage of third-grade students meeting the state standard in reading 
Percentage of third-grade students meeting the state standard in math 
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There arc thus four sets of analysis in the evaluation: 

• Step 1 to Step 2, linking principals’ reported involvement in teacher profes- 
sional development and their receipt of their own professional development 
related to instructional improvement (Chapter 3) 

• Steps 1 and 2 to Step 3, tying teachers’ professional development back first 
to their principal’s involvement and then to the professional development 
that principals themselves received (Chapter 3) 

• Steps 2 and 3 to Step 4, tracing instructional quality back first to teachers’ 
professional development and then to their principal’s involvement in that 
professional development (Chapter 4) 

• Steps 3 and 4 to Step 5, linking student achievement back first to instructional 
quality and then to the professional development that teachers received (Chapter 5) 

The data from teacher surveys and classroom observations at individual schools are ag- 
gregated so that, in the regression analyses, the school is the unit of analysis. 17 In addition, each 
set of analyses is conducted at two or three different levels of specificity. The measures of in- 
structional quality are analyzed both as overall IQA scores and as subscores measuring each 
Principle of Learning. Professional development is considered first with regard to all profes- 
sional development related to improving instructional quality at the school. Then the analysis is 
narrowed to professional development related specifically to the Principles of Learning, first 
combining survey data relating to the three Principles of Learning into one measure and then 
independently analyzing the available data on each Principle of Learning. 

Although it would have been preferable to use a consistent sample across all analyses in or- 
der to strengthen conclusions about relationships that span the various analyses, missing data issues 
made this approach untenable. 18 Therefore, the researchers decided to sacrifice a uniform, but too 
small, sample in the interest of enhancing the meaningfulness of the findings. Consequendy, the 
analyses discussed in this report are generally based on the largest number of schools for which in- 



l7 School means for teacher survey measures were not assigned where fewer than two teachers at the 
school had valid values for the measure, and school means for IQA scores were not assigned where fewer than 
50 percent of the participating third-grade teachers had valid scores. 

ls ln a test of the consistent-sample approach, the sample size was reduced to a point at which the research 
team judged that estimates of the size and significance of the relationships were highly unstable (that is, were 
dependent on the exact composition of the sample) and that meaningful relationships were likely to be ob- 
scured by the lack of statistical power. 
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formation relevant to the analysis is complete. 19 This means that the sample size changes from one 
chapter to another and even from one analysis to another wi t hin the same chapter. 

Since the goal of the analyses is to examine the nature of the relationships between the 
steps in the theory of action independent of other factors that may be expected to influence the 
outcomes, the effects of background characteristics must be accounted for in order to measure 
more accurately the relationships between the variables of interest. For this reason, additional 
covariate measures were included in every analysis, in order to control for certain measurable 
characteristics of principals and teachers and for nonmeasured characteristics of districts. These 
include principals’ length of experience, the average experience of the teachers at the school, 
and indicators for the three school districts in the study. 20 In addition, for the analyses on student 
achievement, a measure of prior school-level third-grade student achievement was included. 
Box 2.3 presents the rationale for including each of these covariates. 

In the following chapters, the findings from the regression analyses are presented in two 
ways. First, check boxes summarize which relationships are statistically significant and which 
are not, at each level of the analysis (for example, the relationships between principals’ receipt 
of and behaviors regarding all instruction-related professional development, professional devel- 
opment on the Principles of Learning, and professional development on each Principle of 
Learning considered separately). Second, bar graphs illustrate the size of certain key relation- 
ships; the bars represent the score on the dependent variable that would be predicted for an indi- 
vidual who received a low, medium, or high score on the independent variable of interest, hold- 
ing constant all other measures that were included in the model. Appendix D presents the full 
results of each regression analysis discussed in this report, including the standardized regression 
coefficient and p-value for each variable included in the model. 



Limitations of the Analysis 

The analyses in this report represent an effort to explore an interesting and suggestive 
theory of change. Understanding the limitations of these analyses helps to make clear the limita- 
tions of the conclusions that can be drawn from them. 



19 There are a few exceptions to this rule: Analyses of Steps 1 through 3 that test the associations of the 
same set of independent variables with two different outcomes (for example, frequency and value of teacher 
professional development) use the same sample for both models. 

20 In order to limit the number of variables included in the regression models, total years of experience as a 
principal was used rather than years of experience as a principal at the study school. The partial correlation 
between a principal’s total experience and the number of years he or she had been at the school (controlling for 
district and average teacher experience) is 0.53. 
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Box 2.3 



Background Characteristics Included and Their Rationale 
as Covariates in the Regression Analyses 

The following control variables are included in all analyses: 

• Principals’ experience 

The number of years of experience that principals have in their position 
is included as a control variable because this may influence their actions 
as instructional leaders. For example, more experienced principals may 
have greater confidence in themselves as leaders, and they may tend to 
hold different beliefs about good teaching practices and the role that 
principals should play in their schools than principals who are newer to 
the job. In addition, more experienced principals may be assigned to 
schools with different characteristics from schools to which less experi- 
enced principals are assigned (for example, experienced principals may 
be assigned to more challenging schools). Finally, a principal’s experi- 
ence was found to be moderately correlated with the number of years 
that the principal reported having been at the same school, which may be 
expected to influence his or her interactions with teachers. 

• Teachers’ experience 

The average experience of the teachers at each school is included as a 
control variable because less experienced teachers may require (or be 
open to) different kinds of assistance from their principals than would 
more experienced teachers. In addition, teachers’ experience may influ- 
ence their teaching practices and as well as the achievement of their stu- 
dents on standardized exams. 

• District 

Indicators for the three study districts are included in each analysis in or- 
der to control for any district-level factors that may shape the behavior of 
principals, the professional development that principals and teachers re- 
ceive, teaching practices, and student achievement levels. 

In addition to the variables listed above, the analyses on student achievement control for one 
additional factor: 

• Prior student achievement 

To account for the school’s history of student achievement, the analyses 
include the percentage of third-grade students at each school who met the 
third-grade performance standard on the state or city assessment for the 
three years prior to the study year (2003, 2004, and 2005). 
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No Counterfactual 

As noted in Chapter 1, no data from non-IFL sites were collected to establish a counter- 
factual (that is, an estimate of what would have happened without the IFL’s presence). Because 
of the absence of such a counterfactual, this report cannot be taken as measuring the impact of 
the IFL’ s work. 

Limited Time Frame 

Ideally, the study would have collected data over time, so that principals’ behavior 
could be linked to changes in instructional quality and student achievement at their schools. But 
because of resource constraints, with the exception of retrospective data on student test scores, 
all the information used in the study was collected during and pertains to the 2005-2006 school 
year. The fact that the data are cross-sectional rather than longitudinal has three major conse- 
quences. 

First, the theory of action informing the report assumes that behaviors will unfold in se- 
quences that have an implicit causal as well as temporal order. But because all the data collected 
for the study are concurrent, it is not possible to determine which actions come first and which 
follow. For this reason, this analysis can provide no empirical evidence as to whether one action 
caused another. For example, the theory postulates that principals will receive professional de- 
velopment on a specific topic and then transmit what they have learned to their teachers (in 
other words, that receipt of professional development causes principals to give their teachers 
professional development on the topic). But the relationship could also be the reverse: Teachers’ 
new knowledge about an instructional approach may make it imperative for principals to learn 
more about that approach. Similarly, the theory suggests that teachers who receive professional 
development on, for example, Accountable Talk and who find it to be valuable will be more 
likely to try out instructional strategies that involve Accountable Talk in their classrooms. But it 
is also possible that teachers will value professional development on a topic because it affirms 
the “rightness” of what they have already been doing in their classes, or that they will value it 
“after the fact” because they observe positive results after implementing in their classrooms the 
teaching practices that they learned in their professional development. While the analyses pre- 
sented in the chapters that follow generally assume the order implied by the theory of action, 
there is no way to be sure that this is correct. 

Furthermore, because the surveys asked only about activities during the 2005-2006 
school year, they do not capture either principals’ or teachers' full exposure to IFL-related pro- 
fessional development, which antedated the study by several years in Austin and Saint Paul and 
by at least one year in Region 10. One might expect principals who received IFL professional 
development over many years to be in a better position to pass it on to their teachers. One would 
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also expect teachers who received IFL professional development over many years to be in a 
better position to put IFL-recommended practices in place in their classrooms. But the data that 
would permit tests of these propositions are not available. 21 

Finally, the theory of action describes a process that unfolds over time. The data avail- 
able for the study compress that process into a one-year time frame that may be too short a pe- 
riod for the changed behaviors predicted by the theory to take effect. For example, a principal 
with strong leadership skills may be assigned to a school where district officials perceive teach- 
ing and learning to be inadequate. In the first year, that principal may bring in expert profes- 
sional development and may improve the instructional climate in other ways. But it would be 
unreasonable to expect an instant turnaround in instructional practices and test scores. In this 
respect, it is notable that, in a review of the effectiveness of comprehensive school reforms 
(CSRs) in improving student achievement, the authors concluded that schools implementing 
such models for five years or more had stronger effects than those with briefer periods of im- 
plementation. 22 

Limited Sample Size 

Another major limitation concerns the small number of principal surveys and the rela- 
tively small numbers of teacher surveys and classroom observations on which the study is 
based. Missing data on particular items or constructs constrain the analyses still further. 22 

As previously noted, the analyses examine schools as the unit of analysis. Most analy- 
ses use the largest sample available to address the key questions, in order to maximize what can 
be learned; the maximum number of schools in any analysis is only 42, while the minimum is 
30. These small overall sample sizes limit the number of variables that can be used to explain 
outcomes in the analyses that follow. Even using the maximum sample available for each 
analysis, the results are not very stable; that is, they are greatly influenced by the specific 
schools included in the analyses, with outliers having considerably more importance than would 
be true in a larger sample. Aside from instability of the estimates, another consequence of the 
small sample size is that relationships between measures that in a larger sample might well be 



“'While, as noted above, principals’ and teachers’ experiences are included in the analyses, these are not 
very good proxies for the amount of IFL training. 

“ 2 Borman, Hewes, Overman, and Brown (2003). 

23 As noted in Chapter 1, the researchers’ interest in testing the general theoiy of action led them to adopt a 
cross-district approach to the analysis. The small sample size is an additional factor in support of that decision. 
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“statistically significant” — that is, not likely to have arisen by chance — are not statistically 
significant in these analyses. 24 

The small number of schools, and of teachers within each school, that were sampled 
further limits the generalizability of the findings. It is impossible to know whether the same pat- 
terns would be observed in other districts where the IFL is working, in other schools in the same 
districts as the study schools, or even in grades other than third grade within the study schools. 

Limitations of Specific Measures 

Finally, there are limitations associated with specific constructs or measures. For exam- 
ple, in order to draw the link between instructional practice and student achievement as tightly 
as possible, it would be ideal to have achievement data available for students within particular 
classrooms within a school, rather than for the school’s entire third grade. The problem of link- 
ing achievement to instructional practice is further complicated by the fact that, as noted in 
Chapter 1, a school was counted as participating in the study if two of its third-grade teachers 
agreed to participate. This means that some schools’ third-grade achievement scores reflect the 
results of instruction delivered by teachers who did not participate in the study and, therefore, 
whose classroom practices could not be observed. 

Similarly, it would have been ideal to base the measures of principals’ instructional 
leadership actions on the survey responses of all teachers in the school, rather than on the re- 
sponses of only the third- and fourth-grade teachers who chose to participate in the study; but 
this was infeasible. Other measurement issues receive attention as they arise in the chapters that 
follow. 



Despite these issues, the Instructional Leadership Study offers a rich lode of data for 
exploration. The analyses in the ensuing chapters take advantage of multiple opportunities to 
link principal surveys, teacher surveys, classroom observations, and achievement data in exam- 
ining a multifaceted and promising theory of how schools improve. 



“‘Because the sample size is so small, the researchers have opted to denote as “statistically significant” 
those differences or relationships that have a 1 in 10 probability or less of occurring by chance. (In statistical 
notation, p < 0.10 [probability is equal to or less than 10 percent].) While it is more conventional to designate 
differences or relationships with a 1 in 20 probability of occurring by chance as statistically significant (p = 
0.05 [5 percent]), it is not obligatory to do so, and a more relaxed standard seems appropriate, given the small 
samples that make it more difficult to identify any relationships as being statistically significant. 
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Chapter 3 

Principals, Teachers, and Professional Development 



This chapter examines the professional development received by principals and teach- 
ers at 44 elementary schools that participated in the Instructional Leadership Study and the con- 
nection of that professional development to instructional leadership (see Box 3.1 for the chap- 
ter’s key findings). 1 The analysis proceeds on the assumption that instructional leaders are 
made, not bom, and that an understanding of the factors that contribute to strong leadership can 
guide districts in training and supervising principals to be more effective heads of schools. 

The areas of Figure 3.1 that are enclosed by a dashed box represent the chapter’s sub- 
ject matter: Steps 1 to 3 of the theory of action developed by the Institute for Learning (IFL) — 
an arm of the Learning Research and Development Center (LRDC) at the University of Pitts- 
burgh — and the survey constructs that are associated with each step. (Chapter 1 describes the 
theory of action, and Appendix C presents the items that make up the survey constructs used in 
the analyses.) Thus, this chapter considers the instruction-oriented professional development 
activities in which principals themselves engaged, as well as principals’ actions in arranging and 
delivering instruction-centered professional development for teachers in their schools and 
teachers’ take-up and reception of it. (As shown in Figure 3.1, many of the report’s exhibits ab- 
breviate “professional development” as “PD.”) 

The discussion takes an expansive view of what constitutes “professional develop- 
ment.” The definition encompasses formal training sessions, to be sure. But it also includes in- 
formal instruction-centered interactions between principals and teachers and among members of 
each group, opportunities for teachers to observe their colleagues either live or on videotape, 
discussions of tests results and student work, and both informal and formal coaching and men- 
toring. The major focus, however, is on those activities that are organized and promoted by dis- 
trict and school administrators, not on self-development efforts that principals or teachers under- 
take on their own. 

The chapter also discusses two additional constructs: teacher’s assessment of the prin- 
cipal as an instructional leader and teacher’s assessment of the principal as an organizational 



'Although there are 49 elementary schools in the study, the analyses in this chapter exclude five schools 
whose principals did not complete the survey (see Chapter 2). Three districts were involved: Austin, Texas; 
Saint Paul, Minnesota; and Region 10 in New York City. For convenience, the report refers to Region 10 as a 
“district.” Beginning in the 2007-2008 school year. New York City adopted a new school organization plan 
that replaced the organization of schools into geographical regions. The study schools served large proportions 
of students who were economically and educationally disadvantaged. 
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Box 3.1 

Key Findings on Instructional Leadership 

• Teachers perceived stronger instructional leaders as stronger organizational lead- 
ers as well. 

• Principals and teachers provided consistent responses about the professional de- 
velopment environments offered by their schools. 

• Principals who were rated higher by their - teachers reported spending more time 
with their teachers on instructional improvement; their schools also provided 
richer learning opportunities for teachers. 

• Both principals and teachers especially valued opportunities to learn from their 
peers. 

• Greater receipt of instruction-related professional development on the part of 
principals and a greater value attached to that professional development are both 
significantly and positively associated with the principals’ involvement in pro- 
fessional development for their teachers. 

• Greater principal involvement in professional development for teachers is sig- 
nificantly and positively associated with the frequency with which teachers re- 
ported receiving professional development. 



leader. The first of these, while not part of the theory of action per se, provides a prism through 
which the remaining constructs are examined. 

The analysis relies on surveys completed by principals and teachers at the study schools 
to address three main questions: 

• How often did principals and teachers receive various kinds of professional 
development, and what did they most value? 

• What did principals who received higher ratings as instructional leaders from 
their teachers do differently than those who received lower ratings? 

• How can the relationships among the steps in the theory of action be ex- 
pressed in quantitative terms? 
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The Instructional Leadership Study 
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NOTE: Constructs referring to professional development on “all Principles of Learning” refer to the Principles of Accountable Talk, Academic Rigor, 
and Clear Expectations, which are the principles that are focused on in this report. 





The next section of the chapter describes findings on the constructs associated with 
Steps 1 to 3 of the theory of action. The subsequent section discusses the process used to arrive 
at an operational definition of instructional leadership and the use of that definition to distin- 
guish three groups of instructional leaders: those who received higher, medium, and lower rat- 
ings from their teachers. The chapter then examines the scores on constructs at each step of the 
theory of action that were achieved by principals in the three leadership categories and by 
teachers at the schools headed by these principals. Collectively, these findings provide an intui- 
tive model of how the theory of action may unfold. That model is then examined formally in a 
section that employs multiple regression analysis to provide quantitative estimates of the link- 
ages among the constructs at the various steps. A discussion of the broader implications of these 
findings for policymakers and administrators is reserved for Chapter 6. 

Box 3.1 summarizes the chapter’s key findings. It is intended to serve as a guidepost for 
readers as they make their way through the discussion that follows. 



The Constructs in Steps 1 to 3 of the Theory of Action 

Table 3.1 shows the mean score, as well as the possible range of scores, on the scale 
measuring each construct associated with Steps 1 to 3. Full statistics on all constructs used in the 
analyses appear in Appendix Table B. 1 . These constructs are discussed below. 

Principals’ Receipt of Professional Development 

It is an IFL tenet that principals need to be knowledgeable about the instructional prac- 
tices that they want their teachers to adopt. They must know what needs to change as well as 
how to bring about that change in order to organize professional development for their teachers 
and to oversee instruction most effectively. District and IFL staff organized formal training ses- 
sions and other kinds of assistance for principals to promote their ability to be instructional 
leaders for their teachers and, in so doing, to improve teaching and learning in their schools. 

The principal survey included questions about the frequency with which various topics 
were covered in professional development opportunities during the summer of 2005 and the 
2005-2006 school year and questions about how valuable the professional development on 
these topics had been. These questions yielded two scales measuring the frequency and value of 
the principal’ s professional development. 2 



“A third scale measured principals’ views of the suppoit that they received from their direct supervisor or 
from their disUict as a whole. The 16-item scale had a reliability (standardized alpha) of 0.919. In theory, a 
principal's score on this scale can range from 1.0 to 4.0; in actuality, the mean score is 3.03, indicating that 
principals were more likely than not to repoit that their supervisors and other district administrators were sup- 
portive. No variables (including principal leadership category or district) were associated with principals' re- 
sponses on the scale, nor did the scale prove useful in further analyses. 
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The Instructional Leadership Study 
Table 3.1 



Selected Constructs Measuring Steps 1 to 3 of the Theory of Action 



Step and Construct 


Mean Score 


Possible Range and Interpretation of Anchor Points 


Step 1: Principal’s receipt of professional development 


Frequency of principal's receipt of 
instruction-related professional de- 
velopment 


1.71 


0 = No instruction-related professional development of any 
type was received. 

4 = Each type of instruction-related professional development 
was received at least once per week. 


Value principal places on instruc- 
tion-related professional develop- 
ment 


3.16 


1 = None of the instruction-related professional development 
received was at all valuable. 

4 = All the instruction-related professional development re- 
ceived was very valuable. 


Step 2: Principal’s involvement in professional development for teachers 


Role principal plays in instruction- 
related professional development of- 
fered to teachers 


2.09 


0 = Principal played no role in any instruction-related profes- 
sional development offered to teachers. 

4.3 = Principal conceptualized, presented, attended, and ar- 
ranged professional development on each topic and attended 
and arranged each type of professional development offered to 
teachers. 


Time principal spends with teachers 
on instructional improvement 


2.91 


1 = Principal rarely or never spent time informally observing 
teachers, reviewing student work or achievement data with 
them, or giving them suggestions on instructional practices. 

4.6 = Principal spent a lot of time (15 or more hours in a typi- 
cal week or at least once a day during a typical school year) on 
each activity. 


Step 3: Teacher’s receipt of professional development 


Frequency of teacher’s receipt of in- 
struction-related professional devel- 
opment 


1.45 


0 = No instruction-related professional development of any 
type was received. 

4 = Each type of instruction-related professional development 
was received at least once per week. 


Value teacher places on instruction- 
related professional development 


3.05 


1 = None of the instruction-related professional development 
received was at all valuable. 

4 = Each kind instruction-related professional development 
received was very valuable. 


Principal's assessment of the 
school's professional development 
environment for teachers 


2.31 


0 = No instruction-related professional development was of- 
fered to teachers, and none of the school’s teachers typically 
participated in any topic or type of professional development. 

4 = Each topic or type of professional development was of- 
fered to teachers at least once a week, and all the school's 
teachers typically participated in each topic or type of profes- 
sional development. 



NOTE: Mean scores were calculated across the 42 schools that are included in the majority of analyses discussed in this 
chapter. 
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Frequency of Principal’s Receipt of Instruction-Related 
Professional Development 

In general, principals reported that they often received professional development related 
to instructional improvement. 3 The mean score of 1.71 on the scale measuring this construct 
suggests that, “on average,” principals received each type of professional development between 
one and three times a year. With just one exception (professional development that involved 
Web-based courses or learning tools), at least 80 percent of the principals reported participating 
at least once or twice during the year in each of the professional development activities included 
in the scale. 

As shown in Box 3.2, some professional development topics were delivered more fre- 
quently than others. It is especially notable that the majority of the principals reported having 
received a good deal of professional development on three of the key IFL Principles of Learn- 
ing: Accountable Talk, Academic Rigor, and Clear Expectations. 4 

Value Principal Places on Instruction-Related Professional Development 

Principals who responded to the survey appreciated the professional development that 
they received. Among those who reported having gotten professional development on the topics 
covered in the survey, some two-thirds or more found each of these activities to be “moderately 
valuable” or “very valuable.” The mean score of 3.16 similarly indicates that principals found 
their professional development to be at least moderately valuable. 



^Specifically, the survey questions measuring frequency of participation asked: “How many times were 
the following topics included in professional development organized by your district for your professional 
growth and development?” and “How often were you offered the opportunity to engage in each of the follow- 
ing types of district-sponsored activities to support your own professional growth and development?" Although 
neither of these questions asked directly how often the respondent had received professional development on a 
particular topic or of a particular type, principals' responses to other survey questions strongly indicate that all 
but a handful of principals interpreted the questions about frequency in terms of activities in which they had 
themselves participated 

4 Chapter 1 and Box 1.1 present the IFL Principles of Learning and describe “Learning Walks” as a profes- 
sional development tool. During Learning Walks, participants spend five to ten minutes in each of several 
classrooms, looking at student work and talking with students and teachers. At the end of the Learning Walk, 
participants work with the leader (generally the principal) to refine observations, look for patterns within the 
school, and identify professional development needs and next steps. Some two-thirds of the principals reported 
that they or their staff members conducted Learning Walks in their own schools at least three times a year. The 
principals were much less likely to be part of Learning Walks in other schools or to report that district staff or 
other principals accompanied them on Learning Walks in their own schools — probably because of the sub- 
stantial advance planning and coordination that Learning Walks require. 

According to interviews with principals and observations of IFL-led professional development sessions. 
Learning Walks aroused considerable trepidation among teachers, who sometimes did not understand that the 
purpose of the activity was not to evaluate their individual teaching practices. 
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Box 3.2 



Most and Least Frequently Included Topics and Types of 
Principals’ Instruction-Related Professional Development 




• Using school-based reading and math curricula to guide instruction 

• Using state and district assessments to guide instruction 

• Helping teachers understand how to communicate their expectations for quality 
work to students 

• Helping teachers understand how to engage students in active reasoning about 
and analysis of challenging content 

• Helping teachers understand how to engage students in whole-group discussions 
to promote learning 

• Helping teachers analyze student performance against the standards 

• Determining what features of student performance need to be improved and how 




• Participating in Learning Walks that district administrative staff and/or other 
principals conduct at your school 

• Participating in Learning walks at other schools 

• Accessing Web-based courses or learning tools 

SOURCES: MDRC calculations from survey responses of 44 principals in study schools in Aus- 
tin, New York City Region 10, and St. Paul. 

NOTES: Italicized topics relate to the IFL Principles of Learning. 

Topics or types of professional development that 70 percent or more of the principals repotted re- 
ceiving at least three times a year. 

^Topics or types of professional development that 40 percent or fewer of the principals reported re- 
ceiving at least three times a year. 
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Box 3.3 shows the professional development topics and/or activities included in the 
scale that principals found most and least valuable. Principals responded especially positively to 
opportunities to learn informally from one another. Over 60 percent of the principals who par- 
ticipated in informal study groups or other types of informal collaboration found these activities 
to be “very valuable.” Principals also highly valued their professional development related to 
the principle of Academic Rigor; their ratings for the professional development related to the 
principles of Clear Expectations and Accountable Talk are somewhat lower. 

Principal’s Involvement in Professional Development for Teachers 

Having received professional development, a major part of what principals must do as 
instructional leaders is to communicate what they have learned to the teachers in their schools. 
The principal survey yielded two scales that measure principals’ involvement in professional 
development for their teachers: role principal plays in professional development offered to 
teachers and time principal spends with teachers on instructional improvement. 

Role Principal Plays in Professional Development Offered to Teachers 

The principal survey asked about various kinds of actions that principals took with re- 
spect to professional development offered to their teachers during the 2005-2006 school year 
and the summer of 2005. With regard to various topics that professional development could 
cover, respondents were asked whether they typically had conceptualized (that is, designed) the 
sessions, presented them, attended sessions presented by someone else, and/or arranged for the 
time, venue, or presenter or whether, alternatively, they had not participated in any way. With 
respect to various forms that the professional development could take, principals were asked 
whether they had attended, arranged, or not participated. In general, the more professional de- 
velopment topics that were covered, the more roles a principal checked off with respect to these 
topics; and the more demanding the roles, the higher the principal’s scale score. 5 

The vast majority of principals reported participating in some way in the professional 
development activities offered to their teachers. As expected, this participation was more likely 
to involve arranging for professional development sessions or attending them than conceptualiz- 
ing or presenting them. Thus, between 52 percent and 83 percent of the principals reported that 
depending on the particular topic or type of activity. In contrast, only 21 percent to 36 percent 
reported that they had conceptualized the sessions. There was one exception: 58 percent of the 



5 Researchers reasoned that conceptualizing or presenting professional development requires more effort 
than arranging or attending it; accordingly, the scale measuring principal's role was scored so that the first two 
responses received two points each, while the latter two received only one point each. Thus, a principal could 
receive as many points for participating in one “harder” activity as in two “easier” ones. 
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Box 3.3 



Topics and Types of Instruction-Related Professional Development 
That Principals Found Most and Least Valuable 




• Helping teachers understand how to engage students in active reasoning about 
and analysis of challenging content 

• Helping teachers analyze student performance against the standards 

• Being coached by another principal 

• Coaching another principal 

• Participating in Learning Walks that district administrative staff and/or other 
principals conduct at your school 

• Informally collaborating or sharing ideas with other principals 

• Participating in an informal study group with other principals 




• Understanding proper implementation of district, state, and federal policies and 
procedures (for example, accountability, attendance, student promotion) 

• Attending district-sponsored/supported university-based programs 

• Attending district-sponsored/supported conferences 

• Accessing Web-based courses or learning tools 

SOURCES: MDRC calculations from survey responses of 44 principals in study schools in Aus- 
tin, New York City Region 10, and St. Paul. 

NOTES: Italicized topics relate to the IFL Principles of Learning. 

Topics or types of professional development that 50 percent or more of the principals reported to 
be “very valuable.” 

^Topics or types of professional development that 30 percent or fewer of the principals reported to be 
“very valuable.” 
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principals reported that they had conceptualized professional development on using students’ 
test results to improve instruction, and 65 percent said that they had presented training on this 
topic. At a time when using data to inform instruction has received much attention — and when 
test results matter greatly for accountability purposes — principals in the study schools have 
heard and responded to the call. 

Time Principal Spends with Teachers on Instructional Improvement 

Other survey questions asked principals to estimate the frequency with which they in- 
teracted with teachers by observing their classes, reviewing student work and achievement data 
with teachers, and giving them suggestions about how to improve instruction. The mean score is 
2.91, indicating that the principal engaged in the “average” activity measured monthly or more 
often. 



Principals spent much more time on some activities than on others. Almost one -third of 
the principals, for example, reported that they spent more than 15 hours a week dropping in on 
teachers to see what was happening in terms of instruction, whereas only 5 percent said that 
they spent a comparable amount of time reviewing student work with teachers. Similarly, half 
of the principals said that they offered teachers general suggestions regarding curriculum and 
instruction several times a week or daily, whereas only 1 1 percent said that they reviewed stu- 
dent achievement data with teachers this often. 

Teachers’ Receipt of Professional Development 

Professional development is widely viewed as a key mechanism for improving instruc- 
tional practice. This section examines the professional development that teachers received from 
the vantage points of both the teachers and their principals. The teacher survey yields data on 
the frequency and value of the professional development that teachers in the study schools re- 
ported receiving. The principal survey contains questions about the frequency with which 
teachers were offered various kinds of professional development and the proportion of teachers 
who participated in professional development activities; together, these questions comprise an 
additional construct: principal’s assessment of the school’s professional development environ- 
ment for teachers. 

Frequency of Teachers’ Receipt of Instruction-Related 
Professional Development 

Like their principals, teachers reported receiving professional development that covered 
a range of subjects and was delivered in different ways. On the whole, however, teachers’ par- 
ticipation in professional development activities was less intense than that of their principals, as 
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reflected by lower mean scores on the scale measuring frequency of receipt of instruction- 
related professional development ( 1 .45 for teachers versus 1.71 for principals). 6 The majority of 
teachers reported receiving some professional development on each of the topics included in the 
scale, but generally only once or twice a year. The professional development that teachers re- 
ceived was more likely to be delivered in formal training sessions and workshops than by less 
conventional means, such as observing other teachers or participating in study groups. Box 3.4 
shows the topics and types of professional development that were most frequently included. 
Notably, teachers reported receiving a good deal of professional development on establishing 
Clear Expectations, one of the three IFL Principles of Learning. 

Value Teacher Places on Instruction-Related Professional Development 

Like principals, teachers believed that the professional development that they received 
was at least moderately valuable, and they rated it almost as highly as the principals rated their 
own professional development (3.05, on average, for teachers, versus 3.16 for principals). 
Teachers rated a few topics or types of professional development as especially valuable or, con- 
versely, as of minimal or no value; these are shown in Box 3.5. Again, it is evident that teachers, 
like principals, especially valued opportunities to work together and learn from one another. 
They also appreciated the professional development that they received on setting clear expecta- 
tions for the quality of student work. On the other hand, a substantial minority (43 percent) of 
teachers who participated in Learning Walks rated them as “minimally valuable” or “not valu- 
able.” 



Principal’s Assessment of the School’s Professional Development 
Environment for Teachers 

This construct combines questions about the frequency with which teachers received 
professional development and the proportion of teachers who participated in these activities. As 
such, it measures the extent to which principals assessed their schools as offering an array of 
professional development opportunities of which teachers took advantage. In theory, scores on 
the scale could range from 0.0 to 4.0; in fact, the mean score across all the schools is 2.31. 

* * * 

The encouraging news of these analyses is that the principals and teachers had similar 
perceptions of their schools’ professional development environments. Schools whose principals 



6 As with the analogous measure for principals, scores on the scale measuring this construct could range 
between 0.0 (indicating that no professional development of any type was received) and 4.0 (indicating that 
each kind of professional development was received at least once a week). 
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Box 3.4 



Most and Least Frequently Included Topics and Types 
of Teachers’ Instruction-Related Professional Development 

Most freauentlv included tvnes/tonics- 



• Aligning curricula and instruction with state and/or district content standards 

• Communicating explicitly to students the expectations for quality work 

• Implementing the reading curriculum 

• Implementing the math curriculum 

• Using student test results to improve/refme instruction 

• Collaborating with other teachers (for example, planning lessons, discussing 
common challenges, analyzing student work) 1 

• Participating in school-based workshops/courses led by an instructional coach or 
other knowledgeable professional' 

Least frequently included types/topics 1 

• Receiving feedback from another teacher who observed your class 

• Participating in content-area study groups in your school 

• Studying video clips to find evidence of effective teaching and learning taking 
place in classrooms 

• Accessing Web-based courses or learning tools 

SOURCES: MDRC calculations from survey responses of 274 third- and fourth-grade teachers in study 
schools in Austin, New York City Region 10, and St. Paul. 

NOTES: Italicized topics relate to the IFL Principles of Learning. 

Topics or types of professional development that 85 percent or more of the teachers reported re- 
ceiving at least once a year. 

topics or types of professional development that 40 percent or more of the teachers reported 
never receiving during the year. 

^Topics or types of professional development that 60 percent or more of the teachers reported re- 
ceiving at least three times a year. 
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Box 3.5 



Topics and Types of Instruction-Related Professional Development 
That Teachers Found Most and Least Valuable 

Most valuable types/topics 1 

• Collaborating with other teachers (for example, planning lessons, discussing 
common challenges, analyzing student work) 

• Communicating explicitly to students the expectation for quality work 

Least valuable types/topics 1 

• Participating in Learning Wa lk s in the teacher’s school 

• Studying video clips to find evidence of effective teaching and learning taking 
place in classrooms 

SOURCES: MDRC calculations from survey responses of 274 third- and fourth-grade teachers in study 
schools in Austin, New York City Region 10, and St. Paul. 

NOTES: Italicized topics relate to the IFL Principles of Learning. 

Topics or types of professional development that 40 percent or more of the teachers who received 
the professional development reported to be “very valuable.” 

Topics or types of professional development that 40 percent or more of the teachers who received 
the professional development repotted to be “minimally valuable” or “not valuable.” 



reported that more teachers participated in professional development activities and did so more 
frequently were, in fact, schools where teachers reported engaging in more frequent professional 
development activities. This similarity of perspectives is, of course, to be hoped for but is by no 
means assured. The surveys administered to principals and to teachers appear to have captured 
aspects of the school environment with reasonable fidelity. 



Steps 1 to 3 Through the Lens of Instructional Leadership 

Defining and Categorizing Instructional Leadership 

One approach to examining the Institute for Learning (IFL) theory of action is to iden- 
tify groups of more or less effective instructional leaders and then to consider the behavior of 
each group with respect to key steps of the theory: receiving professional development and 
passing it along to teachers in their schools. To identify the different leadership groups, the re- 
searchers turned to the teachers, assuming that the latter would be a more accurate source of 
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information about the instructional leadership abilities of their principals than would be the 
principals themselves. The research team developed an instructional leadership scale that re- 
flects the IFL perspective on the behaviors that characterize instructional leaders. 7 The skills 
these items tap fall into four general categories. Good instructional leaders are expected to: 

1. Be generally knowledgeable about and have high expectations of students 
and instruction. 

2. Organize, lead, and attend professional development sessions for teachers in 
their schools. 

3. Conduct Learning Walks and other classroom visits, and provide teachers 
with helpful feedback to improve teaching. 

4. Work with individual teachers and groups of teachers to improve instruction 
in other ways. 

Examples of specific questions that make up the scale derived from the teacher survey 
are shown in Box 3.6; the scale in its entirety is presented in Appendix C.2. Although a few 
items tapped more general attributes of principals, the questions mostly asked teachers about 
specific actions that their principal had performed during the school year. Moreover, while a 
number of questions asked for teachers’ opinions (for example, the extent to which they agreed 
or disagreed with certain statements), the majority of items asked about the frequency with 
which the principal had performed these actions. 

The responses of teachers at a given school to questions about their principal were 
compiled, and an average scale score was calculated for each school. 8 In theory, the mean score 
for a given school could range from 0.45 to 3.91; the mean score across all the schools is 1.97. 9 
Two breaks in the distribution of these scores divided the principals into three groups: 15 prin- 



7 The teacher survey items that constitute the scale were drawn from an unpublished IFL document entitled 
“Hypothesized Effective Leadership Actions of Principals and Supervisors” as well as from a teacher survey 
used in an earlier study of the IFL’s work (see Marsh et al., 2005). 

8 The analyses presented in this section and in the rest of the chapter include 42 schools. As previously 
noted, five of the original 49 schools were excluded because their principals did not respond to the survey. In 
addition, the following analyses exclude one school where only one teacher completed the survey and another 
school whose principal did not answer a background question that is used as a covariate in the analyses. 

9 The scale’s theoretical high point is, however, unrealistic, requiring that principals perform all the speci- 
fied behaviors at least once a week. A more reasonable high score — requiring that principals perform all the 
behaviors monthly — would be 3.36. In actuality, scale scores range from 0.76 to 2.82. 
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Box 3.6 



Examples of Teacher Survey Items Measuring the 
Principal’s Leadership 

Please indicate the extent to which you agree or disagree with the following statements: 
The principal at my school 

. . . sets high standards for teaching and learning. 

. . . understands what students at different grade levels arc expected to know and 
be able to do. 

. . . provides feedback to the faculty on Learning Walks that occur in my school. 

. . . arranges for support when I need it (such as access to coaches, outside con- 
sultants, district curriculum staff). 

Please indicate the frequency with which your principal has 

... led professional development sessions in which you participated. 

. . . attended professional development sessions alongside the staff. 

. . . given you useful feedback and/or suggestions on your teaching. 



ciples received higher ratings from the teachers in their schools; 20 received medium ratings; 
and 8 received lower ratings. 10 



l0 Higher-rated principals had scores of 2.34 or higher; medium-rated principals had scores between 1.61 
and 2.17; and lower-rated principals had scores of 1.48 or lower. 

A school whose principal had a lower rating as an instructional leader was not necessarily devoid of in- 
structional leadership. In some schools, assistant principals and/or coaches undertake some of the instructional 
leadership functions that principals might otherwise handle; in Region 10, assistant principals received much 
the same IFL training as principals, in order to create a leadership team at each school. The study did not col- 
lect data on the activities of individuals other than the principal. However, as discussed below, it appears that 
the principal’s leadership is important in establishing a climate that encourages professional growth: Teachers 
at schools with lower-rated principals reported receiving professional development less frequently than teach- 
ers at schools with higher-rated leaders. 

Multiple regression analysis was used to explore the extent to which various characteristics of principals 
and teachers are associated with leadership category, independent of the other characteristics. The analysis 
indicated that once other variables had been controlled for, only a school's location in Austin is significantly 
associated with leadership score: 1 1 of the 15 schools with stronger leaders are located in that district. 

(continued) 
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The more that teachers reported that their principals adopted the practices comprising 
the measure of instructional leadership, the higher the principal’s leadership score, so it was ex- 
pected that higher-rated principals would be more active than lower-rated ones and would differ 
in other respects as well. That said, the differences among the groups of leaders are sometimes 
sizable, indicating that the scale may effectively distinguish among principals whose behavior 
was, in fact, very different. For example, the bars on the left of Figure 3.2 show that 65 percent 
of teachers at schools with higher-rated leaders — compared with 37 percent of teachers at 
schools with medium-rated leaders and 22 percent of teachers at schools with lower-rated lead- 
ers — strongly agreed that their principal set high standards for teaching and learning. The mid- 
dle bars indicate that 64 percent of the teachers at schools with higher-rated instructional lead- 
ers, 33 percent of the teachers at schools with medium-rated principals, and only 5 percent of 
the teachers at schools with lower-rated principals said that their principal had arranged for 
monthly or more frequent professional development sessions that were relevant to their teaching 
assignments. Finally, the bars on the right show that 35 percent of the teachers at schools with 
higher-rated leadership said that their principal gave useful feedback and/or suggestions on their 
teaching at least once a month, compared with 16 percent of teachers at schools with medium- 
rated leadership and 5 percent of teachers at schools with lower-rated principals. 

Box 3.7 describes in greater detail the leadership behaviors of “Ms. Smith” and “Mr. 
Jones,” two principals (one higher-rated, one lower-rated) whose schools are included in the 
case-study research. 

How Principals in the Three Leadership Groups Scored on the 
Constructs 

The instructional leadership scale was developed using items on the teacher survey. 
But, as it happened, principals in the different leadership categories responded differently to 
items on the principal survey. As Table 3.2 makes clear, the scores of principals in the three 
leadership categories lined up as would be predicted on constructs associated with the first two 
steps in the theory of action. Principals whom teachers rated as stronger instructional leaders 



The data at hand do not explain why Austin principals were more likely to receive higher leadership rat- 
ings than principals at the other two sites. Austin administrators sought to provide school leaders with strong 
instructional leadership skills and encouraged these leaders to transfer what they had learned to their teachers 
— but so did administrators at the other districts in the study. In any case, as noted in Chapter 1, the focus of 
this report is on a theory of action that is intended to describe the experiences of principals and teachers across 
districts. Here, the discussion examines differences among principals in the three leadership categories across 
all three districts, not on the factors that distinguish Austin principals from their colleagues in other sites. Thus, 
in analyses that examine the relationships between leadership rating and other phenomena, district (along with 
principal's experience and teacher’s experience) is used as a control variable. That is, the role of leadership 
rating is considered net of these other variables that could affect the relationships. 
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The Instructional Leadership Study 
Figure 3.2 

Teachers’ Perceptions of Activities of Principals in the 
Three Instructional Leadership Categories 




Teachers strongly agreeing that 
their principal set high 
standards for teaching and 
learning 



Teachers saying that their 
principal arranged for monthly 
or more frequent professional 
development sessions relevant 
to their teaching assignments 



Teachers saying that their 
principal gave useful feedback 
and/or suggestions on their 
teaching at least once a month 



□ Higher-rated instructional leaders □ Medium-rated instructional leaders 

■ Lower-rated instructional leaders 



SOURCE: MDRC calculations from survey responses of 235 teachers in Austin, New York City Region 10, and 
St. Paul. 

NOTE: Percentages are regression-adjusted for principals’ experience, teachers’ experience, and school district. 
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Box 3.7 



Two Case Studies: How a Higher-Rated 
and a Lower-Rated Principal Lead Their Schools 

Ms. Smith and Mr. Jones have both been principals at their schools for three or four years. 
Ms. Smith’s school has more than twice as many classroom teachers as does Mr. Jones’s 
school, but both have high proportions of economically disadvantaged and nonwhite students. 
Neither school is experiencing teacher turnover problems (fewer than 10 percent of the teach- 
ers at each school are new this year), but the third- and fourth-grade teachers at Ms. Smith’s 
school are generally more experienced than those at Mr. Jones’s school (an average of four 
years versus two years as a full-time teacher), and they have generally been at the school 
longer (an average of roughly three years versus two years). 

Ms. Smith received a relatively high instructional leadership scale score from the third- and 
fourth-grade teachers at her school who completed the teacher survey; she was the fifth- 
highest-rated principal in the study. Mr. Jones, on the other hand, received a lower rating on 
the scale from his third- and fourth-grade teachers, which placed him sixth from the bottom of 
the “medium-rated” category discussed in this chapter (nearly two standard deviations below 
Ms. Smith’s rating). But what distinguishes these two principals as instructional leaders? A 
detailed look at the data from the teacher surveys, as well as from principal job-shadows and 
interviews and teacher focus groups conducted at each school, helps to answer this question. 

Looking at the individual teacher survey questions that make up the instructional leadership 
scale, it is immediately apparent that Ms. Smith was consistently reported to engage in a great 
variety of instructional leadership activities (including informally observing her teachers, 
conducting Learning Walks, attending grade-level or content-area meetings, and leading pro- 
fessional development sessions) more frequently than Mr. Jones. A roughly two-and-a-half- 
hour job-shadowing of each principal also revealed impressive differences: Ms. Smith spent a 
total of two hours conducting classroom observations (one hour in each of two classrooms), 
whereas Mr. Jones spent only ten minutes in a classroom during the same time frame. 

Across both the teacher survey and the focus group data, some of the most salient differences 
between the two principals revolve around the nature of their relationships with their teachers 
and in the degree to which they support their teachers in implementing the Principles of 
Learning. Ms. Smith sees herself as a “leader of leaders”; she admits that she cannot be an 
expert in instruction for every grade level and that she depends on experts within the grade 
levels (although it should be noted that all of the teacher survey respondents at her school re- 
ported that she “understands what students at different grade levels arc expected to know and 

(continued) 
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Box 3.7 (continued) 



be able to do,” suggesting that her own assessment may be modest). Teachers at her school 
arc encouraged to present professional development and to learn from each other by partici- 
pating in Learning Wa lk s in the school, which the teachers reported to be exceptionally useful 
for their professional learning. She also expects her teachers to work together in weekly team 
meetings to determine what professional development they need and to agree on modifica- 
tions that should be made to the curriculum to make it better suited to their students’ needs. 
(On the teacher survey, teachers agree that she “engages school staff in developing the 
school’s instructional program.”) She sees one of her primary roles as an instructional leader 
as being to listen to and observe her teachers in order to understand and help meet their needs. 
All the teacher survey respondents agreed that she “arranges for support when [they] need it,” 
and in the focus group each teacher mentioned the supportive and open administration as be- 
ing one of the strengths of the school. However, this does not in any way mean that Ms. 
Smith has a completely hands-off approach to instructional matters in her school. In fact, the 
majority of the teachers responding to the survey at Ms. Smith’s school reported that she gave 
them useful suggestions on how to implement Accountable Talk, Academic Rigor, and Clear 
Expectations in their classrooms at least once a month (which is much more frequently than 
the average principal in the study). Thus, while giving teachers the freedom that they need to 
individualize their own learning and instruction to some extent, she is also highly involved in 
making sure that they arc using the best instructional practices as defined by the IFL. Her 
teachers clearly hold Ms. Smith in great esteem. 

In contrast to Ms. Smith’s open and supportive relationship with her teachers, Mr. Jones re- 
ported that he does not feel that he can trust his teachers to make decisions about instructional 
practice. This could be in part a result of the fact that he has less experienced teachers who 
may not be ready to take on decision-making roles. However, it also appears that Mr. Jones is 
not well aware of his teachers’ needs and is not highly personally involved in their profes- 
sional development.' Only one-third of the teachers who are surveyed reported that he “ar- 
ranges for support when [they] need it,” and in the focus group they commented that they 
needed more professional development in a variety of areas and more support in implement- 
ing the curricula. (On the survey, the majority of his teachers reported that he had not helped 
them with the implementation of the reading or math curricula.) Strikingly, Mr. Jones’s 
teachers also reported in the focus group that the Principles of Learning were never fully ex- 
plained to them (about half of the teachers reported on the survey that Mr. Jones never gave 
them suggestions on how to implement these teaching practices), and they complained that 
they got little feedback from the Learning Wa lk s. Mr. Jones stated that he used the Learning 

(continued) 
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